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(54) IMAGE PROCESSING APPARATUS AND METHOD, AND IMAGE PICKUP APPARATUS 



(57) The present invention relates to an image 
processing device capable of processing of images cor- 
responding to mixing of background images and images 
of moving objects. A region specifying unit 1 03 specifies 
a mixed region made up of a mixture of a foreground 
object component and a background object component, 
and a non-mixed region made up of one or the other of 
a foreground object component and a background ob- 
ject component, and outputs region information corre- 



sponding to the specifying results. A foreground/back- 
ground separation unit 105 separates the input image 
into foreground component images and background 
component images, corresponding to the region infor- 
mation. A separated image processing unit 106 proc- 
esses the foreground component images and back- 
ground component images individually, corresponding 
to the results of separation. The present invention can 
be applied to an image processing device. 
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Description 

Technical Field 

[0001] The present invention relates to an image processing device and method, and an image-taking device, and 
particularly relates to an image processing device and method, and an image-taking device, which take Into consider- 
ation difference between signals detected by sensors and the real world. 

Background Art 

[0002] One type of processing for generating images with higher resolution based on input Images, is class classi- 
fication adaptation processing. An example of class classification adaptation processing is processing wherein coeffi- 
cients used in processing for generating images with higher resolution are generated beforehand, in the spatial direc- 
tion, and images are generated with higher resolution in the spatial direction based on the generated coefficients. 
[0003] Fig. 1 is a block diagram illustrating the configuration of a conventional image processing device for generating 
coefficients used in class classification adaptation processing for generating HD (High Definition) images from SD 
(Standard Definition) images. 

[0004] Frame memory 11 stores input images, which are HD images, in increments of frames. The frame memory 
1 1 supplies the stored HD images to a weighted averaging unit 1 2 and a corresponding pixel obtaining unit 1 6. 
[0005] The weighted averaging unit 12 performs 1/4 weighted averaging on the HD images stored in the frame 
memory 11, generates SD images, and supplies the generated SD images to the frame memory 13. 
[0006] The frame memory 13 stores the SD images supplied from the weighted averaging unit 12 in increments of 
frames, and supplies the stored SD images to a class classification unit 14 and prediction tap obtaining unit 15. . 
[0007] The class classification unit 14 is configured of a class tap obtaining unit 21 and a waveform classification 
unit 22, and performs class classification of pixels of interest which are the pixel of interest in the SD images stored in 
the frame memory 13. The class tap obtaining unit 21 obtains a predetermined number of class taps which are pixels 
of the SD image corresponding to the pixel of interest from the frame memory 13, and supplies the obtained class taps 
to the waveform classification unit 22. 

[0008] Fig. 2 is a diagram explaining the class taps obtained by the class tap obtaining unit 21 . As shown in Fig. 2, 
the class tap obtaining unit 21 obtains eleven class taps at predetermined positions. 

[0009] The waveform classification unit 22 classifies the pixel of interest into one class out of multiple classes, based 
on thedass taps, and supplies a class No. corresponding to the classified class, to the prediction tap obtaining unit 
15. The waveform classification unit 22 classifies the pixel of interest into one class out of 2048 classes, based on the 
eleven class taps. 

[0010] The prediction tap obtaining unit 15 obtains a predetermined number of prediction taps which are pixels of 
the Stf image, corresponding to the classified class from the frame memory 13, based on the class No., and supplies 
the obtained prediction taps and class Nos. to a corresponding pixel obtaining unit 16. 

[001 1] Fig. 3 is a diagram explaining prediction taps which the prediction tap obtaining unit 15 obtains. As shown in 
Fig. 3, the prediction tap obtaining unit 15 obtains nine prediction taps at predetermined locations. 
[0012] The corresponding pixel obtaining unit 16 obtains, from the frame memory 11 , pixels of the HD image corre- 
sponding to the pixel values to be predicted, based on the prediction taps and the class Nos., and supplies the prediction 
taps, class Nos., and the pixels of the HD image corresponding to the obtained pixel values to be predicted, to a normal 
equation generating unit 17. 

[0013] The normal equation generating unit 1 7 generates normal equations corresponding to relationships between 
prediction taps and pixel values to be predicted, corresponding to the classes, . based on the prediction taps, class 
Nos., and the obtained pixel values to be predicted, and supplies the generated normal equations corresponding to 
the classes, to a coefficient calculating unit 18. 

[001 4] The coefficient calculating unit 1 8 solves the normal equation supplied from the normal equation generating 
unit 17, calculates coefficient sets corresponding to each class, and supplies the calculated coefficient sets to coefficient 
set memory 19, along with the class Nos. 

[001 5] The coefficient set memory 1 9 stores the calculated coefficient sets corresponding to the classes, based on 
the class Nos. 

[001 6] Fig. 4 is a diagram explaining an overview of class classification adaptation processing. In class classification 
adaptation processing, a tutor image which is an HD image is used to generate a corresponding SD image, by 1/4 
weighted average processing. The generated SD image is called a student image. 

[001 7] Next, a coefficient set for generating an HD image from the SD image is generated, based on the tutor image 
which is the HD image and the student image which is the corresponding SD image. The coefficient set is configured 
of coefficients for generating an HD image from an SD image, by linear prediction and the like. 
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[0018] A quadruple-density Image is generated from the coefficients set thus generated and the SD image, by linear 
prediction and the like. The processing for generating an image or the like with higher density, from a coefficient set 
and an input image, is also called mapping. 

[0019] SNR comparison, or visual qualitative evaluation is performed, based on the generated quadruple-density 
5 image and a corresponding HD image. 

[0020] A coefficient set generated from a particular tutor image and corresponding student image is called a self 

coefficient set of the particular tutor image and corresponding student image. Mapping using the self coefficient set is 

called self mapping. A coefficient set generated from multiple other tutor images and corresponding student images 

is called a cross coefficient set. 
10 [0021] On the other hand, with images obtained by a video camera taking a foreground subject which moves across 

a predetermined stationary background, movement blurring occurs in the event that the speed of movement of the 

object is relatively fast, and mixing of the foreground and background occurs. 

[0022] With conventional class classification adaptation processing, one set of coefficients is generated for all of the 
foreground, background, and portions where mixing between the foreground and background occurs, by learning 

is processing such as described above, and mapping processing is executed based on the coefficient set. 

[0023] The conventional learning processing for generating coefficients used in the processing for generating HD 
images from SD images will be described, with reference to the flowchart shown in Fig. 6. In Step S11, an image 
processing device judges whether or not there are unprocessed pixels in the student image, and in the event that 
judgment is made that there are unprocessed pixels in the student image, the flow proceeds to Step S12, and pixels 

20 of interest are obtained from the student image, in order of raster scan. 

[0024] In Step S1 3, the class tap obtaining unit 21 of the class classification unit 1 4 obtains a class tap corresponding 
to the pixel of interest, from the student image stored in the frame memory 13. In Step S1 4, the waveform classification 
unit 22 of the class classification unit 14 performs class classification of the pixel of interest, based on the class tap. 
In Step S15, the prediction tap obtaining unit 15 obtains a prediction tap corresponding to the pixel of interest from the 

25 student image stored in the frame memory 13, based on the class into which classification has been made. 

[0025] In Step S16, the corresponding pixel obtaining unit 16 obtains a pixel corresponding to a pixel value to be 
predicted, from tutor data stored in the frame memory 11 , based on the class into which classification has been made. 
[0026] In Step S1 7, the normal equation generating unit 1 7 adds a pixel value of a pixel corresponding to the prediction 
tap and pixel value to be predicted to the matrix for each class, based on the class into which classification has been 

30 made, the flow returns to Step S11 , and the image processing device repeats judgment regarding whether or not there 
are unprocessed pixels. The matrixes for each class to which the pixel value of a pixel corresponding to the prediction 
tap and pixel value to be predicted are added, correspond to the normal equation for calculating coefficients for each 
class. 

[0027] In the event that judgment is made in Step S11 that there are no unprocessed pixels in the student image, 
35 the flcSw proceeds to Step S18, wherein the normal equation generating unit 17 supplies the matrix for each class 
wherein a pixel value of a pixel corresponding to the prediction tap and pixel value to be predicted has been set, to the 
coefficient calculating unit 1 8. The coefficient calculating unit 1 8 solves the matrix for each class wherein a pixel value 
of a pixel corresponding to the prediction tap and pixel value to be predicted has been set, and calculates a coefficient 
set for each class. 

40 [0028] In Step S1 9, the coefficient calculating unit 1 8 outputs the coefficient for each class that has been calculated, 
to the coefficient set memory 1 9. The coefficient set memory 1 9 stores a coefficient set for each class, and the process- 
ing ends. 

[0029] Fig. 7 is a block diagram illustrating the configuration of a conventional image processing device for generating 
HD images from SD images, by class classification adaptation processing. 
45 [0030] Frame memory 31 stores input images which are SD images, in increments of frames. The frame memory 
31 supplies the stored SD images to a mapping unit 32. 

[0031] The SD images input to the mapping unit 32 are supplied to a class classification unit 41 and a prediction tap 
obtaining unit 42. 

[0032] The class classification unit 41 is configured of a class tap obtaining unit 51 and a waveform classification 
50 unit 52, and performs class classification of pixels of interest which are the pixel of interest in the SD images stored in 
the frame memory 31 . The class tap obtaining unit 51 obtains from the frame memory 31 a predetermined number of 
class taps corresponding to the pixel of interest, and supplies the obtained class taps to the waveform classification 
unit 52. 

[0033] The waveform classification unit 52 classifies the pixel of interest into one class out of multiple classes, based 
55 on the class taps, and supplies a class No. corresponding to the classified class, to the prediction tap obtaining unit 42 . 
[0034] The prediction tap obtaining unit 42 obtains from the input image stored in the frame memory 31 a predeter- 
mined number of prediction taps corresponding to the classified class, based on the class No. , and supplies the obtained 
prediction taps and class Nos. to a prediction computation unit 43. 
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[0035] The prediction computation unit 43 obtains coefficient sets corresponding to classes from the coefficient sets 
stored in coefficient set memory 33, based on the class No. The prediction computation unit 43 predicts pixel values 
of predicted images by linear prediction, based on coefficient sets corresponding to classes, and prediction taps. The 
prediction computation unit 43 supplies the predicted pixel values to frame memory 34. 
5 [0036] The frame memory 34 stores predicted pixel values supplied from the prediction computation unit 43, and 
outputs an HD image wherein the predicted pixel values have been set. 

[0037] Fig. 8 is a diagram illustrating the pixel values of the input image, and the pixel values of the output image 
generated by class classification adaptation processing. In Fig. 8, the white squares indicate input signals, and the 
solid circles indicate output signals. As shown in Fig. 8, the image generated by the class classification adaptation 
<v 10 processing contains waveforms lost in the bandwidth restriction of the SD image. In this sense, it can be said that 

processing for generating an image with higher resolution by the class classification adaptation processing creates 
resolution. 

[0038] The conventional processing for creating images, for generating HD images from SD image with an image 
processing device which executes processing for creating resolution by class classification adaptation processing, will 

15 be described with reference to the flowchart in Fig. 9. 

[0039] In Step S31, the image processing device judges whether or not there are unprocessed pixels in the input 
image, and in the event that judgment is made that there are unprocessed pixels in the input image, the flow proceeds 
to Step S32, where the mapping unit 32 obtains a coefficient set stored in the coefficient set memory 33. In Step S33, 
the image processing device obtains pixels of interest from the input image in raster scan order. 

20 [0040] In Step S34, the class tap obtaining unit 51 of the class classification unit 4 1 obtains a class tap corresponding 
to the pixel of interest, from the input image stored in the frame memory 31 . In Step S35, the waveform classification 
unit 52 of the class classification unit 41 performs class classification of the pixel of interest into one class, based on 
the class tap. 

[0041] In Step S36, the prediction tap obtaining unit 42 obtains a prediction tap corresponding to the pixel of interest 
25 from the input image stored in the frame memory 31 , based on the class into which classification has been made. 

[0042] In Step S37, the prediction computation unit 43 obtains a pixel value of a predicted image by linear prediction, 

based on the coefficient set corresponding to the class into which classification has been made, and the prediction tap. 

[0043] In Step S38, the prediction computation unit 43 outputs the predicted pixel value to the frame memory 34. 

The frame memory 34 stores the pixel value supplied from the prediction computation unit 43. The procedures return 
30 to Step S31 , and repeats judgement regarding whether or not there are unprocessed pixels. 

[0044X In the event that judgment is made in Step S31 that there are no unprocessed pixels in an input image, the 

flow proceeds to Step S39, where the frame memory 34 outputs the stored predicted image wherein predicted values 

are set, and the processing ends. 

[0045] Also, edge enhancing processing is used for converting the input image into an image with the sense-of- 
35 resolution enhanced even further. As with the class classification adaptation processing described above, the same 
processing is executed for the entire screen with the edge enhancement processing as well. 
[0046] However, in the event that objects move in front of still backgrounds, movement blurring occurs not only due 
to mixture of the moving object images itself, but also due to mixture of the moving object images and the background 
images. Conventionally, processing images corresponding to the mixing of the background image and the image of 
<o the moving object had not been given thought. 



Disclosure of Invention 



[0047] The present invention has been made in light of the above, and it is an object thereof to enable processing 
45 of images corresponding to the mixing of background images and images of the moving objects. 

[0048] An image processing device according to the present invention comprises: region specifying means for spec- 
ifying, based on the input image data, one or the other of a mixed region made up of a mixture of a foreground object 
component configuring foreground objects and a background object component configuring background objects, and 
a non-mixed region made up of one of a foreground region made up of the foreground object component and a back- 
so ground region made up of a background object component configuring the background objects, and outputting region 
specifying information corresponding to the results of specifying; separating means for separating the input image data 
in at least the mixed region into the foreground object component and the background object component, corresponding 
to the region specifying information ; and processing means for individually processing the foreground object component 
and the background object component, corresponding to the results of separation. 
55 [0049] The image processing device may further comprise removing means for removing movement blurring of at 
least one of the foreground object component and the background object component, with the processing means 
individually processing the foreground object component and the background object component which have been 
subjected to movement blurring removal. 
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[0050] The region specifying means may further specify a covered background region and an uncovered background 
region, and output the region specifying information corresponding to the results of specifying, with the separating 
means separating the input image data into the foreground object component and the background object component 
in the covered background region and the uncovered background region. 

[0051 ] The processing means may generate coefficients used for class classification adaptation processing, for each 
of the foreground object component and the background object component. 

[0052] The processing means may generate output image data for each of the foreground object component and 
the background object component, by class classification adaptation processing. 

[0053] The processing means may perform edge enhancement for each of the foreground object component and 
the background object component. 

[0054] The image processing device may further comprise: 

foreground component image generating means for generating a foreground component image by synthesizing the 
foreground object component separated in the mixed region and the pixel data of the foreground region; and background 
component image generating means for generating a background component image by synthesizing the background 
object component separated in the mixed region and the pixel data of the background region; with the processing 
means individually processing the foreground component image and the background component image which are 
generated. 

[0055] An image processing method according to the present invention comprises: a region specifying step for spec- 
ifying, based on the input image data, one or the other of a mixed region made up of a mixture of a foreground object 
component configuring foreground objects and a background object, component configuring background objects, and 
a non-mixed region made up of one of a foreground region made up of the foreground object component and a back- 
ground region made up of a background object component configuring the background objects, and outputting region 
specifying information corresponding to the results of specifying; a separating step for separating the input image data 
in at least the mixed region into the foreground object component and the background object component, corresponding 
to the region specifying information; and a processing step for individually processing the foreground object component 
and the background object component, corresponding to the results of separation. 

[0056] The image processing method may further comprise a removing step for removing movement blurring of at 
least one of the foreground object component and the background object component, with the foreground object com- 
ponent and the background object component which have been subjected to movement blurring removal being indi- 
vidually processed in the processing step. 

[0057] In the region specifying step, a covered background region and an uncovered background region may be 
further specified, and the region specifying information corresponding to the results of specifying output, with the input 
image data being separated into the foreground object component and the background object component in the covered 
background region and the uncovered background region in the separating step. 

[0056} In the processing step, coefficients used for class classification adaptation processing may be generated, for 
each of the foreground object component and the background object component. 

[0059] In the processing step, output image data may be generated for each of the foreground object component 
and the background object component, by class classification adaptation processing. 

[0060] in the processing step, edge enhancement may be performed for each of the foreground object component 
and the background object component. 

[0061] The image processing method may further comprise: a foreground component image generating step for 
generating a foreground component image by synthesizing the foreground object component separated in the mixed 
region and the pixel data of the foreground region; and a background component image generating step for generating 
a background component image by synthesizing the background object component separated in the mixed region and 
the pixel data of the background region; with the foreground component image and the background component image 
which are generated, being individually processed in the processing step. 

[0062] A program in a recording medium according to the present invention comprises: a region specifying step for 
specifying, based on the input image data, one or the other of a mixed region made up of a mixture of a foreground 
object component configuring foreground objects and a background object component configuring background objects, 
and a non-mixed region made up of one of a foreground region made up of the foreground object component and a 
background region made up of a background object component configuring the background objects, and outputting 
region specifying information corresponding to the results of specifying; a separating step for separating the input 
image data in at least the mixed region into the foreground object component and the background object component, 
corresponding to the region specifying information; and a processing step for individually processing the foreground 
object component and the background object component, corresponding to the results of separation. 
[0063] The program in the recording medium may further comprise a removing step for removing movement blurring 
of at least one of the foreground object component and the background object component, with the foreground object 
component and the background object component which have been subjected to movement blurring removal being 
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individually processed in the processing step. 

[0064] In the region specifying step, a covered background region and an uncovered background region may be 
further specified, and the region specifying information corresponding to the results of specifying output, with the input 
image data being separated into the foreground object component and the background object component in the covered 
background region and the uncovered background region in the separating step. 

[0065] In the processing step, coefficients used for class classification adaptation processing may be generated, for 
each of the foreground object component and the background object component. 

[0066] In the processing step, output image data may be generated for each of the foreground object component 
and the background object component, by class classification adaptation processing. 

[0067] In the processing step, edge enhancement may be performed for each of the foreground object component 
and the background object component. 

[0068] The program in the recording medium may further comprise: a foreground component image generating step 
for generating a foreground component image by synthesizing the foreground object component separated in the mixed 
region and the pixel data of the foreground region; and a background component image generating step for generating 
a background component image by synthesizing the background object component separated in the mixed region and 
the pixel data of the background region; with the foreground component image and the background component image 
which are generated, being individually processed in the processing step. 

[0069] A program according to the present invention causes a computer to execute: a region specifying step for 
specifying, based on the input image data, one or the other of a mixed region made up of a mixture of a foreground 
object component configuring foreground objects and a background object component configuring background objects, 
and a non-mixed region made up of one of a foreground region made up of the foreground object component and a 
background region made up of a background object component configuring the background objects, and outputting 
region specifying information corresponding to the results of specifying; a separating step for separating the input 
image data in at least the mixed region into the foreground object component and the background object component, 
corresponding to the region specifying information; and a processing step for individually processing the foreground 
object component and the background object component, corresponding to the results of separation. 
[0070] The program may further comprise a removing step for removing movement blurring of at least one of the 
foreground object component and the background object component, with the foreground object component and the 
background object component which have been subjected to movement blurring removal being individually processed 
in the processing step. 

[0071] in the region specifying step, a covered background region and an uncovered background region may be 
further specified, and the region specifying information corresponding to the results of specifying output, with the input 
image data being separated into the foreground object, component and the background object component in the cov- 
ered background region and the uncovered background region in the separating step. 

[0072) In the processing step, coefficients used for class classification adaptation processing may be generated, for 
each of the foreground object component and the background object component. 

[0073] in the processing step, output image data may be generated for each of the foreground object component 
and the background object component, by class classification adaptation processing. 

[0074] In the processing step, edge enhancement may be performed for each of the foreground object component 
and the background object component. 

[0075] The program may further comprise: a foreground component image generating step for generating a fore- 
ground component image by synthesizing the foreground object component separated in the mixed region and the 
pixel data of the foreground region; and a background component image generating step for generating a background 
component image by synthesizing the background object component separated in the mixed region and the pixel data 
of the background region; with the foreground component image and the background component image which are 
generated, being individually processed in the processing step. 

[0076] An image-taking device according to the present invention comprises: image-taking means for outputting a 
subject image taken by an image-taking device having a predetermined number of pixels having time- integration effects 
as taken image data made up of a predetermined number of pieces of pixel data; region specifying means for specifying, 
based on the taken image data, one or the other of a mixed region made up of a mixture of a foreground object 
component configuring foreground objects and a background object component configuring background objects, and 
a non-mixed region made up of one of a foreground region made up of the foreground object component and a back- 
ground region made up of a background object component configuring the background objects, and outputting region 
specifying information corresponding to the results of specifying; separating means for separating the taken image 
data in at least the mixed region into the foreground object component and the background object component, corre- 
sponding to the region specifying information; and processing means for individually processing the foreground object 
component and the background object component, corresponding to the results of separation. 
[0077] The image-taking device may further comprise removing means for removing movement blurring of at least 
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one of the foreground object component and the background object component, with the processing means individually 
processing the foreground object component and the background object component which have been subjected to 
movement blurring removal. 

[0078] The region specifying means may further specify a covered background region and an uncovered background 
region, and output the region specifying information corresponding to the results of specifying, with the separating 
means separating the taken image data into the foreground object component and the background object component 
in the covered background region and the uncovered background region. 

[0079] The processing means may generate coefficients used for class classification adaptation processing, for each 
of the foreground object component and the background object component. 

[0080] The processing means may generate output image data for each of the foreground object component and 
the background object component, by class classification adaptation processing. 

[0081] The processing means may perform edge enhancement for each of the foreground object component and 

the background object component. 

[0082] The image-taking device may further comprise: 

foreground component image generating means for generating a foreground component image by synthesizing the 
foreground object component separated in the mixed region and the pixel data of the foreground region; and background 
component image generating means for generating a background component image by synthesizing the background 
object component separated in the mixed region and the pixel data of the background region, with the processing 
means individually processing the foreground component image and the background component image which are 
generated. 

[0083] One or the other of a mixed region made up of a mixture of a foreground object component configuring fore- 
ground objects and a background object component configuring background objects, and a non-mixed region made 
up of one of a foreground region made up of the foreground object component and a background region made up of 
a background object component configuring the background objects, are specified, region specifying information cor- 
responding to the results of specifying is output, the input image data in at least the mixed region is separated into the 
foreground object component and the background object component, corresponding to the region specifying informa- 
tion, and the foreground object component and the background object component are individually processed, corre- 
sponding to the results of separation. 

[0084] Thus, in the event that a moving object is photographed, for example, images can be processed corresponding 
to the mixing of background images and moving object images. 

Brief description of the Drawings 

[0085] 

Ffg. 1 is a block diagram illustrating the configuration of a conventional image processing device. 
Fig. 2 is a diagram explaining class taps. 
Fig. 3 is a diagram explaining prediction taps. 

Fig. 4 is a diagram describing the overview of class classification adaptation processing. 
Fig. 5 is a diagram explaining conventional coefficient sets. 
Fig. 6 is a flowchart explaining conventional learning processing. 

Fig. 7 is a block diagram illustrating the configuration of a conventional image processing device. 

Fig. 8 is a diagram illustrating pixel values of an output image generated by pixel values of an input image, and 

class classification adaptation processing. 

Fig. 9 is a flowchart explaining conventional processing for creating images. 

Fig. 1 0 is a block diagram illustrating the configuration of an embodiment of an image processing device according 
to the present invention. 

Fig. 11 is a block diagram illustrating a configuration of an image processing device. 

Fig. 12 is a diagram describing image-taking by a sensor. 

Fig. 13 is a diagram describing an arrangement of pixels. 

Fig. 14 is a diagram describing operation of a detecting device. 

Fig. 15A is a diagram describing an image obtained by taking an image of an object corresponding to a moving 
foreground, and an object corresponding to a still background. 

Fig. 15B is a diagram describing a model corresponding to an image obtained by taking an image of an object 
corresponding to a moving foreground, and an object corresponding to a still background. 
Fig. 16 is a diagram describing background region, foreground region, mixed region, covered background region, 
and uncovered background region. 

Fig. 17 is a model diagram which develops, over the time direction, the pixel values of pixels adjacently arrayed 
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in one row, in an image wherein an object corresponding to a still foreground and an object corresponding to a still 
background are subjected to image-taking. 

Fig. 1 8 is a model diagram wherein the pixel values are developed over the time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 1 9 is a model diagram wherein the pixel values are developed over the time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 20 is a model diagram wherein the pixel values are developed over the time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 21 is a diagram illustrating an example of extracting pixels of the foreground region, background region, and 
mixed region. 

Fig. 22 is a diagram illustrating how pixels correspond to a model wherein pixel values are developed over the 
time direction. 

Fig. 23 is a model diagram wherein the pixel values are developed over the time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 24 is a model diagram wherein the pixel values are developed over the time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 25 is a model diagram wherein the pixel values are developed over the time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 26 is a model diagram wherein the pixel values are developed over the time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 27 is a model diagram wherein the pixel values are developed over the time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 28 is a diagram illustrating the correlation between a divided image, and a model diagram wherein the pixel 

values of pixels are developed over the time direction. 

Fig. 29 is a diagram illustrating an example of a divided image. 

Fig. 30 is a diagram illustrating an example of a divided image. 

Fig. 31 is a flowchart explaining the processing of images with the image processing device according to the 
present invention. 

Fig. 32 is a block diagram illustrating an example of the configuration of the region specifying unit 103. 
Fig. 33 is a diagram describing an image wherein an object corresponding to the foreground is moving. 
Fig. 34 is a model diagram wherein the pixel values are developed over the time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 35 is a model diagram wherein the pixel values are developed over the time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 36 is a model diagram wherein the pixel values are developed overthe time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 37 is a diagram describing conditions for region judgment. 

Fig. 38A is a diagram illustrating an example of the results of region specification made by the region specifying 
unit 103. 

Fig. 38B is a diagram illustrating an example of the results of region specification made by the region specifying 
unit 103. 

Fig. 38C is a diagram illustrating an example of the results of region specification made by the region specifying 
unit 103. 

Fig. 38D is a diagram illustrating an example of the results of region specification made by the region specifying 
unit 103. 

Fig. 39 is a diagram illustrating an example of the results of region specification made by the region specifying unit 
103. 

Fig. 40 is a flowchart explaining processing for region specifying. 

Fig. 41 is a block diagram illustrating another example of the configuration of the region specifying unit 103. 
Fig. 42 is a model diagram wherein the pixel values are developed overthe time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 43 is a diagram illustrating an example of a background region image. 

Fig. 44 is a block diagram illustrating the configuration of a binary object image extracting unit 302. 

Fig. 45A is a diagram describing calculating of correlation values. 

Fig. 45B is a diagram describing calculating of correlation values. 

Fig. 46 A is a diagram describing calculating of correlation values. 

Fig. 46B is a diagram describing calculating of correlation values. 

Fig. 47 is a diagram illustrating an example of a binary object image. 
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Fig. 48 is a block diagram illustrating the configuration of a time change detecting unit 303. 
Fig. 49 is a diagram describing judgment of a region judgment unit 342. 

Fig. 50 is a diagram illustrating an example of judgment made by the time change detecting unit 303. 

Fig. 51 is a flowchart describing processing for region specification by the region judgment unit 1 03. 

Fig. 52 is a flowchart for describing the processing for region specification in detail. 

Fig. 53 is a block diagram illustrating yet another configuration of the region specifying unit 103. 

Fig. 54 is a block diagram describing the configuration of a robustrfication unit 361. 

Fig. 55 is a diagram describing movement compensation of a movement compensation unit 381 . 

Fig. 56 is a diagram describing movement compensation of a movement compensation unit 381 . 

Fig. 57 is a flowchart describing the processing for region specification. 

Fig. 58 is a flowchart describing details of processing for robustification. 

Fig. 59 is a block diagram illustrating an example of the configuration of a mixture ratio calculating unit 104. 
Fig. 60 is a diagram illustrating an example of an ideal mixture ratio a. 

Fig. 61 is a model diagram wherein the pixel values are developed over the time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 62 is a model diagram wherein the pixel values are developed over the time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 63 is a diagram describing approximation using correlation of foreground components. 
Fig. 64 is a diagram describing the relation between C, N, and P. 

Fig. 65 is a block diagram illustrating another configuration of the estimated mixture ratio processing unit 401 . 

Fig. 66 is a diagram illustrating an example of an estimated mixture ratio. 

Fig. 67 is a block diagram illustrating another configuration of a mixture ratio calculation unit 104. 

Fig. 68 is a flowchart explaining the processing for calculating mixture ratio. 

Fig. 69 is a flowchart describing processing for computing an estimated mixture ratio. 

Fig. 70 is a diagram describing a straight line approximating a mixture ratio a. 

Fig. 71 is a diagram describing a plane approximating a mixture ratio a. 

Fig. 72 is a diagram describing how pixels in multiple frames correspond at the time of calculating the mixture ratio a. 
Fig. 73 is a block diagram illustrating another configuration of the mixture ratio estimation processing unit 401 . 
Fig. 74 is a diagram illustrating an example of an estimated mixture ratio. 
Fig. 75 is a flowchart explaining the processing for calculating mixture ratio. 

Fig. 76 is a flowchart describing the processing for mixture ratio estimation by way of a model corresponding to a 
covered background region. 

Fig. 77 is a block diagram illustrating an example of the configuration of a foreground/background separation unit 
105. 

Ffg. 78A is a diagram illustrating an input image, foreground region image, background region image, foreground 
component image, and background component image. 

Fig. 78B is a model diagram corresponding to an input image, foreground region image, background region image, 
foreground component image, and background component image. 

Fig. 79 is a model diagram wherein the pixel values are developed over the time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 80 is a model diagram wherein the pixel values are developed over the time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 81 is a model diagram wherein the pixel values are developed over the time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 82 is a block diagram illustrating an example of the configuration of the separating unit 601 . 

Fig. 83 is a flowchart describing the processing for separating the foreground and the background. 

Fig. 84 is a block diagram illustrating the configuration of a separated image processing unit 106 which generates 

coefficient sets. 

Fig. 85 is a diagram explaining the relation between a tutor image and a student image. 

Fig. 86 is a block diagram illustrating the configuration of a learning unit 14. 

Fig. 87A is a diagram explaining class classification processing. 

Fig. 87B is a diagram explaining class classification processing. 

Fig. 88A is a diagram explaining ADRC processing. 

Fig. 88B is a diagram explaining ADRC processing. 

Fig. 89 is a diagram explaining coefficient sets which the separated image processing unit 106 generates. 

Fig. 90 is a flowchart explaining the learning processing for generating coefficient sets with the separated image 

processing unit 106. 

Fig. 91 is a flowchart for explaining the processing for generating coefficient sets corresponding to the background 



EP1 396 819 A1 



region. 

Fig. 92 is a block diagram illustrating the configuration of the separated image processing unit 1 06 which generates 

a higher resolution image in the spatial direction by executing class classification adaptation processing. 

Fig. 93 is a block diagram illustrating the configuration of a mapping unit 807. 

Fig. 94A is a diagram illustrating an example of an image In the mixed region of a tutor image. 

Fig. 94B is a diagram illustrating change in pixel values corresponding to the position in the spatial direction of an 

image in the mixed region of a tutor image. 

Fig. 95A is a diagram illustrating an example of an image in a mixed region, generated by conventional class 
classification adaptation processing. 

Fig. 95B is a diagram illustrating change in the pixel values corresponding to position in the spatial direction of an 
image in a mixed region, generated by conventional class classification adaptation processing. 
Fig. 96A is a diagram illustrating an example of an image in a mixed region, generated by the image processing 
device according to the present invention. 

Fig. 96B is a diagram illustrating change in the pixel values corresponding to the position in the spatial direction 

of a mixed region image, generated by the image processing device according to the present invention. 

Fig. 97A is a diagram illustrating an example of an image in a foreground region of a tutor image. 

Fig. 97B is a diagram illustrating change in pixel values of an image In the foreground region of a tutor image. 

Fig. 98A is a diagram illustrating an example of an image in a foreground region, generated by conventional class 

classification adaptation processing. 

Fig. 98B is a diagram illustrating change in the pixel values of an image in a foreground region, generated by 
conventional class classification adaptation processing. 

Fig. 99A is a diagram illustrating an example of an image in a foreground region, generated by the image processing 
device according to the present invention. 

Fig. 99B is a diagram illustrating change in the pixel values corresponding to position in the spatial direction of a 

foreground region image, generated by the image processing device according to the present invention. 

Fig. 1 00 is a flowchart explaining the processing for creating images with the separated image processing unit 106. 

Fig. 101 is a flowchart describing processing for predicting images corresponding to the background region. 

Fig. 1 02 is a block diagram illustrating the configuration of the separated image processing unit 1 06 wherein edge 

enhancing processing with difference effects is applied for each region. 

Fig. 103 is a block diagram illustrating the configuration of an edge enhancing unit 907, 

Fig. 104A is a diagram describing the processing for edge enhancement. 

Fig. 104B is a diagram describing the processing for edge enhancement. 

Fig. 1 04C is a diagram describing the processing for edge enhancement. 

Fig. 105 is a diagram illustrating filter coefficients. 

Fig. 106 is a diagram explaining operation of a highpass filter 921 . 

Fig. 107 is a diagram illustrating filter coefficients. 

Fig. 108 is a diagram explaining operation of the highpass filter 921 . 

Fig. 109 is a block diagram illustrating another configuration of the edge enhancing unit 907. 

Fig. 110 is a diagram illustrating filter coefficients. 

Fig. 111 is a diagram explaining operation of a filter 941 . 

Fig. 112 is a diagram illustrating filter coefficients. 

Fig. 113 is a diagram explaining operation of the filter 941 . 

Fig. 114 is a diagram explaining the processing by the separated image processing unit 106. 

Fig. 115 is a flowchart explaining the processing of edge enhancement processing with the separated image 

processing unit 106. 

Fig. 116 is a block diagram illustrating another configuration of the functions of the image processing device. 
Fig. 11 7 is a block diagram illustrating an example of the configuration of a mixture ratio calculating unit 11 01 . 
Fig. 11 8 is a block diagram illustrating an example of the configuration of a foreground/background separation unit 
1102. 

Fig. 11 9 is a block diagram illustrating another configuration of the functions of the image processing device. 
Fig. 120 is a diagram illustrating the correlation between a divided image, and a model diagram wherein the pixel 
values of pixels are developed over the time direction. 

Fig. 1 21 is a diagram illustrating the correlation between an image wherein movement blurring has been removed, 
and a model diagram wherein the pixel values of pixels are developed over the time direction. 
Fig. 122 is a diagram describing processing of the image processing device according to the present invention. 
Fig, 123 is a flowchart explaining the processing of images with the image processing device according to the 
present invention. 

Fig. 1 24 is a block diagram illustrating an example of the configuration of a foreground/background separation unit 
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2001. 

Fig. 125A is a diagram illustrating an input image, foreground component image, and background component 
image. 

Fig. 125B is a model diagram corresponding to an input image, foreground component image, and background 
component image. 

Fig. 126 is a model diagram wherein the pixel values are developed over the time direction, and periods corre- 
sponding to shutter time are divided. 

Fig. 127 is a model diagram wherein the pixel values are developed over the time direction, and periods corre- 
sponding to shutter time are divided. 

Fig. 128 is a model diagram wherein the pixel values are developed over the time direction, and periods corre- 
sponding to shutter time are divided. 

Fig. 129 is a block diagram illustrating an example of the configuration of a separating unit 2601 . 

Fig. 130A is a diagram illustrating an example of a separated foreground component image. 

Fig. 130B is a diagram illustrating an example of a separated background component image. 

Fig. 131 is a flowchart describing the processing for separating the foreground and the background. 

Fig. 132 is a block diagram illustrating an example of the configuration of a movement blurring removal unit 2002. 

Fig. 133 is a diagram describing increments of processing. 

Fig. 1 34 is a model diagram wherein the pixel values of foreground component image are developed over the time 
direction, and periods corresponding to shutter time are divided. 

Fig. 135 is a model diagram wherein the pixel values of foreground component image are developed over the time 
direction, and periods corresponding to shutter time are divided. 

Fig. 136 is a model diagram wherein the pixel values of foreground component image are developed over the time 
direction, and periods corresponding to shutter time are divided. 

Fig. 137 is a flowchart explaining processing for removing movement blurring contained in the foreground compo- 
nent image by the movement blurring removal unit 2002. 
Fig. 138 is a diagram illustrating a model of a background component image. 
Fig. 139 is a diagram illustrating a model of a corrected background component image. 

Fig, 140 is a block diagram illustrating the configuration of the movement-blurring-removed-image processing unit 
2004 which generates coefficient sets. 

Fig. 141 is a block diagram illustrating the configuration of a learning unit 3006. 

Fig. 142 is a diagram explaining coefficient sets which the movement-blurring-removed-image processing unit 
2004 generates. 

Fig. 1 43 is a flowchart explaining the learning processing for generating coefficient sets by the movement-blurring- 
removed-image processing unit 2004. 

Fig. 1 44 is a flowchart for explaining the processing for generating coefficient sets corresponding to the background 
component image. 

Fig. 145 is a block diagram illustrating the configuration of the movement-blurring-removed-image processing unit 
2004 which executes class classification adaptation processing and generates a higher resolution image in the 
spatial direction. 

Fig. 146 is a diagram illustrating a model of a foreground component image wherein movement blurring has been 
removed. 

Fig. 147 is a diagram illustrating a model of a foreground component image wherein movement blurring has been 
added. 

Fig. 148 is a block diagram illustrating the configuration of a mapping unit 3103. 

Fig. 149 is a flowchart explaining the processing for creating an image with regard to the movement-blurring- 
removed-image processing unit 2004. 

Fig, 150 is a flowchart describing processing for predicting images corresponding to the background component 
image. 

Fig. 151 is a block diagram illustrating the configuration of the movement-blurring-removed-image processing unit 
2004 wherein edge enhancing processing with difference effects is applied for each image. 
Fig. 152 is a diagram explaining the processing of the movement-blurring-removed-image processing unit 2004. 
Fig. 153 is a flowchart explaining the processing of edge enhancement processing with the movement-blurring- 
removed-image processing unit 2004. 

Fig. 154 is a block diagram illustrating another configuration of the functions of the image processing device. 
Fig. 155 is a block diagram illustrating an example of the configuration of a foreground/background separation unit 
501. 
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Best Mode for Carrying Out the Invention 

[0086] Fig. 1 0 is a block diagram which illustrates the configuration of an embodiment of the image processing device 
according to the present invention. A CPU (Central Processing Unit) 71 performs various types of processing following 
programs stored in ROM (Read Only Memory) 72, or a storage unit 78. RAM (Random Access Memory) 73 suitably 
stores programs for the CPU 71 to execute, data, and so forth. These CPU 71 , ROM 72, and RAM 73 are mutually 
connected via a bus 74. 

[0087] The CPU 71 is also connected to an input/output interface 75 via the bus 74. The input/output interface 75 is 
connected to an input unit 76 such as a keyboard, mouse, microphone, or the like, and is connected to an output unit 
77 such as a display, speaker, or the like. The CPU 71 performs various types of processing corresponding to instruc- 
tions input from the input unit 76. The CPU 71 then outputs images, audio, or the like, which are obtained as a result 
of processing, to the output unit 77. 

[0088] The storage unit 78 connected to the input/output interface 75 comprises a hard disk, for example, and stores 
programs for the CPU 71 to execute and various types of data. A communication unit 79 communicates with external 
devices via the Internet or other networks. In this case of the example, the communication unit 79 serves as an obtaining 
unit which obtains output from a sensor. 

[0089] Also, an arrangement may be made wherein programs are obtained via the communication unit 79, and are 
stored in the storage unit 78. 

[0090] A drive 80 connected to the input/output interface 75 drives a magnetic disk 91, optical disk 92, magneto- 
optical disk 93, semiconductor memory 94, or the like, in the event that those are mounted thereon, and obtains pro- 
grams and data stored therein. The obtained programs and data are transmitted to the storage unit 78 and stored 
therein, as necessary. 

[0091] Fig. 11 is a block diagram which illustrates the configuration of the functions of the image processing device 
according to the present invention. 

[0092] Note that whether each function of the image processing device is realized by hardware or software does not 
matter. That is to say, each block diagram in the present Specification may be regarded as not only a hardware block 
diagram but also as a software function block diagram. 

[0093] Here, the input image input in the image processing device contains movement blurring. 
[0094] The movement blurring means distortion which is included in images corresponding to moving objects, which 
occurs due to movement of objects which are objects of image-taking in the real world and due to image-taking prop- 
erties of the sensor. 

[0095]- In the present Specification, images corresponding to objects which are objects of image-taking in the real 
world are called image objects. 

[0096] Input images provided to the image processing device are provided to an object extracting unit 1 01 , a region 

specifying unit 1 03, a mixture ratio calculating unit 104, and a foreground/background separation unit 105. 

[0097] The object extracting unit 101 roughly extracts the image objects corresponding to the foreground object 

contained in the input image, and supplies the extracted image object to the movement detecting unit 102. The object 

extracting unit 1 01 roughly extracts the image object corresponding to the foreground object, for example, by detecting 

the outlines of the image object corresponding to the foreground object contained in input image. 

[0098] The object extracting unit 101 roughly extracts the image object corresponding to the background object 

contained in the input image, and supplies the extracted image object to the movement detecting unit 102. The object 

extracting unit 1 01 roughly extracts the image object corresponding to the background object, by the difference between 

the input image and the image object corresponding to the extracted foreground object, for example. 

[0099] Also, for example, an arrangement may be made wherein the object extracting unit 101 roughly extracts the 

image objects corresponding to the foreground objects and the image objects corresponding to the background objects 

based upon the difference between the background images stored in background memory provided therein and the 

input images. 

[01 00] The movement detecting unit 1 02 calculates the movement vectors of the image object corresponding to the 
roughly extracted foreground objects by techniques such as block matching, gradation, phase correlation, pixel recur- 
sion, or the like, and supplies the calculated movement vectors and movement vector position information (information 
for specifying the pixel positions corresponding to the movement vectors) to the region specifying unit 103. 
[0101] The movement vector output from the movement detecting unit 1 02 includes information corresponding to a 
movement amount v. 

[0102] Also, for example, an arrangement may be made wherein the movement detecting unit 1 02 outputs the move- 
ment vector per image object to the movement blurring adjustment unit 1 06 along with the pixel position information 
for specifying a pixel of the image object. 

[0103] The movement amount v is a value which represents the change of position of the image corresponding to 
the moving object in increments of pixel interval. For example, in the event that the object image corresponding to the 
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foreground moves so as to be displayed at a position four pixels away in the following frame with a given frame as a 
reference, the movement amount v of the image of the object corresponding to the foreground is 4. 
[0104] The region specifying unit 1 03 classifies each pixel of the input image into one of the foreground region, the 
background region, or the mixed region which consists of the uncovered background region and the covered back- 

5 ground region, and supplies the information which indicates which of the foreground region, the background region, 
or the mixed region which consists of the uncovered background region or the covered background region, each pixel 
belongs to, (which will be referred to as region information hereafter), to the mixture ratio calculation unit 104 and the 
foreground/background separation unit 1 05. Note that details of the mixed region, uncovered background region, and 
the covered background region will be described later. 

10 [0105] The mixture ratio calculating unit 1 04 calculates the mixture ratio corresponding to the pixels contained in the 
mixed region (which will be referred to as mixture ratio a hereafter) based upon the input image and the region infor- 
mation supplied from the region specifying unit 103, and supplies the calculated mixed ratio to the foreground/back- 
ground separating unit 105. 

[0106] The mixture ratio a is a.value which represents the ratio of the image component corresponding to the back- 
's ground object (which will also be referred to as background component hereafter) with the pixel value as indicated in 
Expression (3) described below. 

[0107] The foreground/background separation unit 1 05 separates the image components corresponding to the fore- 
ground object (which will be also referred to as the foreground component hereafter) and the background component 
image which consists of only the background components based upon the region information supplied from the region 
20 specifying unit 1 03 and the mixtu re ratio a supplied from the mixture ratio calculation unit 1 04, and supplies the image 
in the background region, the image which consists of only the background components in the uncovered background 
region (which will be referred to as the background component image in the uncovered background region), the image 
which consists of only the foreground components in the uncovered background region (which will be referred to as 
the foreground component image in the uncovered background region), the image which consists of only the back- 
us ground components in the covered background region (which will be referred to as the background component image 
in the covered background region), the image which consists of only the foreground components in the covered back- 
ground region (which will be referred to as the foreground component image in the covered background region), and 
the image in the foreground region, to the separated image processing unit 106. 

[0108] The separated image processing unit 106 performs processing for the image in the background region, the 
30 background component image in the uncovered background region, the foreground component image in the uncovered 
background region, the background component image in the covered background region, the foreground component 
image in the covered background region, and the image in the foreground region, supplied from the foreground/back- 
ground separation unit 105, respectively. 

[0109] For example, the separated image processing unit 106 generates coefficients which are used in the class 
35 classification adaptation processing for generating an even higher resolution image for each of the image in the back- 
ground region, background component image in the uncovered background region, foreground component image in 
the uncovered background region, background component image in the covered background region, foreground com- 
ponent image in the covered background region, and image in the foreground region. 

[01 10] For example, the separated image processing unit 1 06 creates an even higher resolution image by applying 
40 the class classification adaptation processing for each of the image in the background region, background component 
image in the uncovered background region, foreground component image in the uncovered background region, back- 
ground component image in the covered background region, foreground component image in the covered background 
region, and image in the foreground region. 

[0111] Also, for example, the separated image processing unit 106 applies the processing for edge enhancement 
45 with differing degrees by using a different coefficient for each of the image in the background region, the background 
component image in the uncovered background region, the foreground component image in the uncovered background 
region, the background component image in the covered background region, the foreground component image in the 
covered background region, and the image in the foreground region, 

[0112] The input images supplied to the image processing device will now be described, referring to Fig. 12 through 
so Fig. 27. 

[01 13] Fig. 1 2 is a diagram which describes image-taking with a sensor. The sensor comprises a CCD video camera 
or the like, for example, including a CCD (Charge-Coupled Device) area sensor which is a solid-state image-taking 
device. An object 111 corresponding to the foreground in the real world moves between an object 112 corresponding 
to the background in the real world, and the sensor, for example, from the left side to the right side horizontally in the 
55 drawing. 

[01 1 4] The sensor takes images of the object 1 1 1 corresponding to the foreground with the object 1 1 2 corresponding 
to the background. The sensor outputs the taken images in increments of one frame. For example, the sensor outputs 
images of 30 frames per second. In this case, the exposure period of the sensor is 1/30 seconds. The exposure period 
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represents a period from the sensor beginning conversion of input light into electric charges, up to the end of conversion 
of input light to electric charges. The exposure period will be also referred to as a shutter period hereafter. 
[01 1 5] Fig. 1 3 is a diagram which describes an arrangement of pixels. In Rg. 1 3, A through I denote individual pixels. 
These pixels are arranged on a plane corresponding to the image. One detecting element corresponding to one pixel 
is disposed on the sensor. Upon the sensor taking images, one detecting element outputs pixel values corresponding 
to one pixel which makes up the image. For example, a position In the X direction of the detecting elements corresponds 
to a position in the horizontal direction on the image, and a position in the Y direction of the detecting elements corre- 
sponds to a position in the vertical direction on the image. 

[01 16] As shown in Fig. 1 4, for example, the detecting element of the CCD converts the input light into charges for 
a period corresponding to the shutter period, and accumulates the converted charges. The quantity of charges is 
approximately proportional to the strength of the input light and the period during which the light is input. The detecting 
element adds the charges converted from the input light to the accumulated charges in the period corresponding to 
the shutter period. That is to say, the detecting element integrates the input light during the period corresponding to 
the shutter period, and accumulates the amount of charges corresponding to the integrated light. It can also be said 
that the detecting element has integrating effects with regard to time. 

[01 1 7] The charges accumulated in the detecting element are converted into a voltage value by a circuit not shown 
in the drawings, which is further converted to pixel values such as digital data or the like, and is output. Accordingly, 
individual pixel values output from a sensor have values projected in one-dimensional space, which is from a result 
wherein a given portion having a spatial extension of the object corresponding to the foreground or the background, 
is integrated for the shutter period. 

[0118] The image processing device extracts valid information buried in output signals due to such accumulation 
operations of the sensor, such as the mixture ratio a, for example. 

[01 1 9] Fig. 1 5A and Fig. 1 5B are diagrams which describe the image which is obtained by taking image of the object 
corresponding to the moving foreground and the object corresponding to the still background. Fig. 15A illustrates the 
image which is obtained by taking image of the object corresponding to the foreground with movement and the object 
corresponding to the still background. With the example shown in Fig. 1 5A, the object corresponding to the foreground 
moves from the left to the right horizontally in the drawing. 

[0120] Fig. 15B is a model diagram wherein pixel values corresponding to one line of the image shown in Fig. 15A 
develop over the time direction. The horizontal direction in Fig. 15B corresponds to the spatial direction X in Fig. 15A. 
[0121] The pixel values of pixels in the background regions are made up of only the background components, i.e., 
the image components corresponding to the background objects. The pixel values of pixels in the foreground regions 
are made up of only the foreground components, i.e., the image components corresponding to the foreground objects. 
[0122] The pixel values of pixels in mixed regions are made up of the background components and the foreground 
components. Since the pixel values in the mixed region consists of the background components and the foreground 
compenents, the mixed region can also be said to be a distortion region. The mixed regions are further classified into 
covered background regions and uncovered background regions. 

[0123] The covered background region is a mixed region at a position corresponding to a leading portion in the 
progress direction of the foreground object with regard to the foreground region, and accordingly is a region wherein 
the background components are covered by the foreground corresponding to elapsing of time. 
[0124] Conversely, the uncovered background region is a mixed region at a position corresponding to a trailing portion 
in the progress direction of the foreground object with regard to the foreground region, and accordingly is a region 
wherein the background components emerge corresponding to elapsing of time. 

[0125] As described above, images including the foreground region, background region, covered background region, 
or uncovered background region, are input as input images to the region specifying unit 1 03, the mixture ratio calculating 
unit 104, and the foreground/background separation unit 105. 

[0126] Fig. 16 is a diagram which describes the background region, foreground region, mixed region, covered back- 
ground region, and uncovered background region, as described above. In the event of corresponding to the images 
shown in Fig. 1 5A, the background region is the still portion, the foreground region is the moving portion, the covered 
background region of the mixed region is the portion which changes from the background to the foreground, and the 
uncovered background region of the mixed region is the portion which changes from the foreground to the background. 
[0127] Fig. 1 7 is a model diagram wherein pixel values of the pixels arrayed adjacently in one line in the image that 
has been taken of the objects corresponding to the still foregrounds and the objects corresponding to the still back- 
grounds, develop over the time direction. For example, pixels arrayed in one line in a screen may be selected, as pixels 
adjacently arrayed in one line. 

[0128] The pixel values F01 through F04 shown in Fig. 17 are pixel values of pixels corresponding to the still fore- 
ground object. The pixel values B01 through B04 shown in Fig. 17 are pixel values of pixels corresponding to the still 
background object. 

[0129] The vertical direction in Fig. 17 represents elapsing of time from the top to the bottom in the drawing. The 
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position of the upper side of the rectangle in Fig. 1 7 corresponds to the time at which the sensor begins conversion of 
the input light into charges, and the position of the lower side of the rectangle in Fig. 17 corresponds to the time at 
which the sensor ends the conversion of the input light into charges. That is to say, the distance from the upper side 
to the lower side of the rectangle in Fig. 1 7 corresponds to the shutter period. 

[0130] An arrangement wherein the shutter period equals the frame interval will now be described below, by way of 
an example. 

[0131] The horizontal direction in Fig. 1 7 corresponds to the spatial direction X as described in Fig. 15A. More par- 
ticularly, shown by way of an example in Fig. 17, the distance from the left side of the rectangle denoted by "F01 a to 
the right side of the rectangle denoted by n B04" in Fig. 1 7, is eight times long as the pixel pitch, that is to say, corresponds 
to the interval of eight continuous pixels. 

[0132] In the event that the foreground objects and the background objects keep still, the light input to the sensor is 
not altered during the period corresponding to the shutter period. 

[0133] Now, the period corresponding to the shutter period is divided into two or more periods of equal length. For 
example, in the event that the virtual dividing number is 4, the model diagram shown in Fig. 1 7 can be represented by 
the model shown in Fig. 1 8. The virtual dividing number is set corresponding to the movement amount v or the like of 
the object corresponding to the foreground within the shutter period. For example, corresponding to the movement 
amount v of 4, the virtual dividing number is 4, and the period corresponding to the shutter period is divided into 4 
periods. 

[01 34] The uppermost row in the drawing corresponds to the first of the divided periods from the shutter being opened. 
The second row from the top in the drawing corresponds to the second of the divided periods from the shutter being 
opened. The third row from the top in the drawing corresponds to the third of the divided periods from the shutter being 
opened. The fourth row from the top in the drawing corresponds to the fourth of the divided periods from the shutter 
being opened. 

[0135] The divided shutter period corresponding to the movement amount v is also referred to as a shutter period/ 
v hereafter. 

[0136] In the event that the object corresponding to the foreground keeps still, the foreground component F01/V 
equals the value in which the pixel value F01 is divided by the virtual dividing number, since the light input to the sensor 
is not altered. Similarly, in the event that the object corresponding to the foreground keeps still, the foreground com- 
ponent F02/v equals the value of the pixel value F02 being divided by the virtual dividing number, the foreground 
component F03/v equals the value of the pixel value F03 being divided by the virtual dividing number, and the foreground 
component F04/v equals the value of the pixel value F04 being divided by the virtual dividing number. 
[0137] In the event that the object corresponding to the background keeps still, the background component B01/v 
equals the value of the pixel value B01 being divided by the virtual dividing number, since the light input to the sensor 
is not altered. Similarly, in the event that the object corresponding to the background keeps still, the background com- 
ponent B02/V equals the value of the pixel value B02 being divided by the virtual dividing number, B03/v equals the 
value of the pixel value B03 being divided by the virtual dividing number, and B04/v equals the value of the pixel value 
B04 being divided by the virtual dividing number. 

[0138] That is to say, in the event that the object corresponding to the foreground keeps still, the foreground com- 
ponent F01/v corresponding to the first shutter period/v from the shutter opening, the foreground component F01/v 
corresponding to the second shutter period/v from the shutter opening, the foreground component F01 /v corresponding 
to the third shutter period/v from the shutter opening, and the foreground component F01/V corresponding to the fourth 
shutter period/v from the shutter opening, are the same value, since the light corresponding to the foreground object 
which is input to the sensor is not altered during the period corresponding to the shutter period. F02/v through F0AN 
have the same relationship as F01/v. 

[0139] In the event that the object corresponding to the background keeps still, the background component B01/V 
corresponding to the first shutter period/v from the shutter opening, the-background components B01 /v corresponding 
to the second shutter period/v from the shutter opening, the background components B01/v corresponding to the third 
shutter period/v from the shutter opening, and the background components B01/V corresponding to the fourth shutter 
period/v from the shutter opening, are the same value, since the light corresponding to the background object which 
is input to the sensor is not altered during the period corresponding to the shutter period. B02/v through B04/v have 
the same relationship. 

[0140] A case will now be described wherein the object corresponding to the foreground moves while the object 
corresponding to the background keeps still. 

[0141] Fig. 19 is a model diagram wherein pixel values of the pixels on one line including the covered background 
region deveiop over the time direction in the event that, the object corresponding to the foreground moves towards the 
right side in the drawing. In Fig. 19, the movement amount v of the foreground is 4. Since one frame is a short period, 
an assumption may be made that the object corresponding to the foreground is a rigid body, and moves at a constant 
velocity. 
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[0142] In Fig. 19, the object image corresponding to the foreground moves so as to be displayed at a position four 
pixels to the right in the following frame, with a given frame as a reference. 

[01 43] In Fig. 1 9, the left-most pixel through the fourth pixel from the left, belong to the foreground region. In Fig. 1 9, 
the fifth through the seventh pixels from the left belong to the covered background region of the mixed region. In Fig. 
19, the right-most pixel belongs to the background region. 

[01 44] Since the object corresponding to the foreground moves so as to hide the object corresponding to the back- 
ground with elapsing of time, the components contained in the pixel values of the pixels which belong to the covered 
background region change from the background components to the foreground components at a certain point of the 
period corresponding to the shutter period. 

[0145] For example, the pixel value M shown with a heavy frame in Fig. 19, is represented by Expression (1). 

M = B02/V -»- B02/V + F07/V + F06/v (1 ) 

[01 46] For example, since the fifth pixel from the left includes a background component corresponding to one shutter 
period/v and foreground components corresponding to the three shutter period/vs, the mixture ratio a of the fifth pixel 
from the left is 1/4. Since the sixth pixel from the left contains background components corresponding to the two shutter 
period/vs and foreground components corresponding to the two shutter period/vs, the mixture ratio a of the sixth pixel 
from the left is 1/2. Since the seventh pixel from the left includes background components corresponding to the three 
shutter period/vs and a foreground component corresponding to the one shutter period/v, the mixture ratio a of the 
seventh pixel from the left is 3/4. 

[0147] Since an assumption may be made that the object corresponding to the foreground is a rigid body and the 
foreground image moves at a constant velocity so as to be displayed at a position four pixels to the right in the following 
frame, the foreground component F07/v of the first shutter period/v from the shutter opening of the fourth pixel from 
the left in Fig. 19, for example, equals the foreground component corresponding to the second shutter period/v from 
the shutter opening of the fifth pixel from the left in Fig. 19. Similarly, the foreground component F07/v equals the 
foreground component corresponding to the third shutter period/v from the shutter opening of the sixth pixel from the 
left in Fig. 19, and the foreground component corresponding to the fourth shutter period/v from the shutter opening of 
the seventh pixel from the left in Fig. 19, respectively. 

[0148] Since an assumption may be made that the object corresponding to the foreground is a rigid body and that 
the foreground image moves at a constant velocity so as to be displayed at a point four pixels to the right in the following 
frame, the foreground component F06/v of the first shutter period/v from the shutter opening of the third pixel from the 
left in Fig. 19, for example, equals the foreground component corresponding to the second shutter period/v from the 
shutter opening of the fourth pixel from the left in Fig. 19. Similarly, the foreground component F06/v equals the fore- 
ground component corresponding to the third shutter period/v from the shutter opening of the fifth pixel from the left in 
Fig. 19, and the foreground component corresponding to the fourth shutter period/v from the shutter opening of the 
sixth pixel from the left in Fig. 19, respectively. 

[0149] . Since an assumption may be made that the object corresponding to the foreground is a rigid body and the 
foreground image moves at a constant velocity so as to be displayed at a position four pixels to the right in the following 
frame, the foreground component F05/V of the first shutter period/v from the shutter opening of the second pixel from 
the left in Fig. 19, for example, equals the foreground component corresponding to the second shutter period/v from 
the shutter opening of the third pixel from the left in Fig. 19. Similarly, the foreground component F05/v equals the 
foreground component corresponding to the third shutter period/v from the shutter opening of the fourth pixel from the 
left in Fig. 1 9, and the foreground component corresponding to the fourth shutter period/v from the shutter opening of 
the fifth pixel from the left in Fig. 19, respectively. 

[0150] Since an assumption may be made that the object corresponding to the foreground is a rigid body and the 
foreground image moves at a constant velocity so as to be displayed at a position four pixels to the right in the following 
frame, the foreground component F04/v of the first shutter period/v from the shutter opening of the left-most pixel in 
Fig. 1 9, for example, equals the foreground component corresponding to the second shutter period/v from the shutter 
opening of the second pixel from the left in Fig. 19. Similarly, the foreground component F04/v equals the foreground 
component corresponding to the third shutter period/v from the shutter opening of the third pixel from the left in Fig. 
1 9, and the foreground component corresponding to the fourth shutter period/v from the shutter opening of the fourth 
pixel from the left in Fig. 19, respectively. 

[0151] As described above, the foreground region corresponding to the moving object includes movement blurring, 
so this can be said to be a distorted region. 

[0152] Fig. 20 is a model diagram wherein the pixel values of the pixels on one line including the uncovered back- 
ground region develop over the time direction in the event that the foreground moves toward the right side in the 
drawing. In Fig. 20, the movement amount v of the foreground is 4. Since one frame is a short time, an assumption 
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may be made that the object corresponding to the foreground is a rigid body, and moves at a constant velocity. In Fig. 
20, the object image corresponding to the foreground moves to the right side by four pixels in the following frame with 
a given frame as a reference. 

[0153] In Fig. 20, the left-most pixel through tne fourth pixel from the left, belong to the background region. In Fig. 
20, the fifth through the seventh pixels from the left belong to the mixed region of the uncovered background. In Fig. 
20, the right-most pixel belongs to the foreground region. 

[0154] Since the object corresponding to the foreground which has hidden the object corresponding to the back- 
ground moves so as to be removed from the front of the object corresponding to the background with elapsing of time, 
the components included in the pixel values of the pixels which belong to the uncovered background region change 
from the foreground components to the background components at a certain point in the period corresponding to the 
shutter period. 

[0155] For example, the pixel value M* indicated with a heavy frame in Fig. 20, is represented by Expression (2). 



[0156] For example, since the fifth pixel from the left includes the background components corresponding to the 
three shutter period/vs, and the foreground component corresponding to the one shutter period/v, the mixture ratio a 
of the fifth pixel from the left is 3/4. Since the sixth pixel from the left includes the background components corresponding 
to the two shutter period/vs and the foreground components corresponding to the two shutter period/vs, the mixture 
ratio a of the sixth pixel from the left is 1/2. Since the seventh pixel from the left includes the background component 
corresponding to the one shutter period/v and the foreground components corresponding to the three shutter period/ 
vs, the mixture ratio a of the seventh pixel from the left is 1/4. 

[0157] Further generalizing Expression (1) and Expression (2), the pixel value M is represented by Expression (3). 



[01 58] Here, a denotes the mixture ratio. B denotes the pixel value of the background, and Fi/v denotes the foreground 
component. 

[0159] Since an assumption may be made that the object corresponding to the foreground is a rigid body and moves 
at a constant velocity, and the movement amount v is 4, for example, the foreground component F01/V of the first 
shutter period/v from the shutter opening of the fifth pixel from the left in Fig. 20 equals the foreground component 
corresponding to the second shutter period/v from the shutter opening of the sixth pixel from the left in Fig. 20. Similarly, 
F01/V equals the foreground component corresponding to the third shutter period/v from the shutter opening of the 
seventh pixel from the left in Fig. 20, and the foreground component corresponding to the fourth shutter period/v from 
the shutter opening of the eighth pixel from the left in Fig. 20, respectively. 

[01 60] Since an assumption may be made that the object corresponding to the foreground is a rigid body and moves 
at a constant velocity, and the virtual dividing number is 4, the foreground component F02/v of the first shutter period/ 
v from the shutter opening of the sixth pixel from the left in Fig. 20, for example, equals the foreground component 
corresponding to the second shutter period/v from the shutter opening of the seventh pixel from the left in Fig. 20. 
Similarly, the foreground component F02/v equals the foreground component corresponding to the third shutter period/ 
v from the shutter opening of the eighth pixel from the left in Fig. 20. 

[01 61 ] Since an assumption may be made that the object corresponding to the foreground is an rigid body and moves 
at a constant velocity, and movement amount v is 4, the foreground component F03/v of the first shutter period/v from, 
the shutter opening of the seventh pixel from the left in Fig. 20, for example, equals the foreground component corre- 
sponding to the second shutter period/v from the shutter opening of the eighth pixel from the left in Fig. 20. 
[0162] While a description has been made in the description of Fig. 18 through Fig. 20 wherein the virtual dividing 
number is 4, the virtual dividing number corresponds to the movement amount v. The movement amount v generally 
corresponds to the movement velocity of the object corresponding to the foreground. For example, in the event that 
the object corresponding to the foreground moves so as to be displayed at a position four pixels to the right in the 
following frame with a given frame as a reference, the movement amount v is 4. The virtual dividing number is 4 
corresponding to the movement amount v. Similarly, for example, in the event that the object corresponding to the 
foreground moves so as to be displayed at a position six pixels to the left in the following frame with a given frame as 
a reference, the movement amount v is 6, and the virtual dividing number is 6. 

[0163] Fig. 21 and Fig. 22 illustrate the relationship between the foreground region, the background region, and the 



M' = F02/V + F01/V + B26/V + B26N 



(2) 




(3) 
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mixed region which consists of the covered background region orthe uncovered background region, and the foreground 
components and the background components, corresponding to the divided shutter period. 
[0164] Fig. 21 illustrates an example wherein the pixels of the foreground. region, the background region, and the 
mixed region, are extracted from the image including the foreground corresponding to the object which moves in front 
5 of the still background. The reference character A shown in Fig. 21 denotes an object which moves in front of the still 
background. In the example shown in Fig. 21 , the object corresponding to the foreground denoted by the reference 
character A moves horizontally with regard to the screen. 

[0165] The frame #n + 1 is the frame following the frame #n, and the frame #n + 2 is the frame following the frame 
#n + 1. 

w [0166] Fig. 22 illustrates a model wherein the pixels of the foreground region, the background region, and the mixed 
region are extracted from one of frame #n through frame #n + 2, and the pixel values of the extracted pixels are 
developed over the time direction, with the movement amount v at 4. 

[0167] Since the object corresponding to the foreground moves, the pixel values of the foreground region consist of 
four different foreground components corresponding to the period of shutter period/v. For example, the left-most pixel 
15 of the pixels of the foreground region shown in Fig. 22 consists of F01/V, F02/v, F03/v, and F04/V. That is to say, the 
pixels of the foreground region include movement blurring. 

[0168] Since the object corresponding to the background keeps still, the light corresponding to the background input 
to the sensor is not altered during the period corresponding to the shutter period. In this case, the pixel values of the 
background region do not contain movement blurring. 

20 [0169] The pixel value of the pixel which belongs to the mixed region made up of the covered background region or 
the uncovered background region consists of the foreground components and the background components. 
[0170] Next, a model will be described wherein, in the event that the image corresponding to the object moves, the 
pixel values of the pixels which are arrayed adjacently in a single line on multiple frames, and at the same position in 
the frames, develop over the time direction. For example, in the event that the image corresponding to the object moves 

25 horizontally on the screen, the pixels arrayed in a single line can be selected as pixels arrayed adjacently in a single line. 
[0171] Fig. 23 is a model diagram wherein the pixel values of pixels arrayed adjacently in a single line on threef rames 
of images which are taken of the object corresponding to the still background, and are at the same position in the 
frames, develop over the time direction. The frame #n is the frame following the frame #n - 1 , and the frame #n + 1 is 
the frame following the frame #n. Other frames are denoted in the same way. 

30 [0172] The pixel values of the B01 through B12 shown in Fig. 23 are the pixel values of the pixels corresponding to 
the object of the still background. Since the object corresponding to the background keeps still, the pixel values of the 
corresponding pixels do not change in the frame #n - 1 through the frame #n + 1 . For example, the pixels in the frame 
#n and the pixels in the frame #n + 1 at the position corresponding to the pixel having a pixel value B05 in the frame 
#n - 1 , have a pixel value B05, respectively. 

35 [01 73j Fig. 24 is a model diagram wherein the pixel values of pixels arrayed adjacently in a single line on three frames 
of images taken of the object corresponding to the foreground which moves to the right side in the drawing with the 
object corresponding to the still background, and at the same position in the frames, develop over the time direction. 
The models shown in Fig. 24 includes the covered background region. 

[0174] Since an assumption may be made in Fig. 24 that the object corresponding to the foreground is a rigid body 
40 and moves at a constant velocity, and the foreground image moves so as to be displayed at a position four pixels to 
the right side in the following frame, the foreground movement amount v is 4, and the virtual dividing number is 4. 
[01 75] For example, the foreground component of the first shutter period/v from the shutter opening of the left-most 
pixel of the frame #n - 1 in Fig. 24 is F1 2/v, the foreground component of the second shutter period/v from the shutter 
opening of the second pixel from the left in Fig. 24 is also F12/V. The foreground component of the third shutter period/ 
45 v from the shutter opening of the third pixel from the left in Fig. 24, and the foreground component of the fourth shutter 
period/v from the shutter opening of the fourth pixel from the left in Fig. 24, are F12/v. 

[0176] The foreground component of the second shutter period/v from the shutter opening of the left-most pixel in 
the frame #n - 1 in Fig. 24 is F1 1 /v, and the foreground component of the third shutter period/v from the shutter opening 
of the second pixel from the left in Fig. 24 is also F11/v. The foreground component of the fourth shutter period/v from 

50 the shutter opening of the third pixel from the left In Fig. 24 is F11/v. 

[0177] The foreground component of the third shutter period/v from the shutter opening of the left-most pixel in the 
frame #n - 1 in Fig. 24 is F1 ON, and the foreground component of the fourth shutter period/v from the shutter opening 
of the second pixel from the left in Fig. 24 is also F1 0/v. The foreground component of the fourth shutter period/v from 
the shutter opening of the left-most pixel in the frame #n - 1 in Fig. 24 is F09/v. 

55 [0178] Since the object corresponding to the background keeps still, the background component of the first shutter 
period/v from the shutter opening of the second pixel from the left in the frame #n - 1 in Fig. 24 is B01/v. The background 
components of the first and second shutter period/vs from the shutter opening of the third pixel from the left in the 
frame #n - 1 in Fig. 24 are B02/v. The background components of the first through third shutter period/vs from the 
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shutter opening of the fourth pixel from the left in the frame #n - 1 in Fig. 24 are B03/v. 

[0179] In the frame #n - 1 in Fig. 24, the left-most pixel belongs to the foreground region, and the second through 
fourth pixels from the left belong to the mixed region of the covered background region. 

[0180] The fifth through twelfth pixels from the left in the frame #n - 1 in Fig. 24 belong to the background region, 

5 and the pixel values thereof are B04 through B1 1 , respectively. 

[0181] The first through fifth pixels from the left in the frame #n in Fig. 24 belong to the foreground region. The 
foreground component of the shutter period/v in the foreground region in the frame #n, is one of F05/v through F12/v. 
[0182] Since an assumption may be made that the object corresponding to the foreground is a rigid body and moves 
at a constant velocity, and the foreground image moves so as to be displayed at a position four pixels to the right side 

w in the following frame, the foreground component of the first shutter period/v from the shutter opening of the fifth pixel 
from the left in the frame #n in Fig. 24 is F12/v, the foreground component of the second shutter period/v from the 
shutter opening of the sixth pixel from the left in Fig. 24 is also F12/v. The foreground component of the third shutter 
period/v from the shutter opening of the seventh pixel from the left in Fig. 24, and the foreground component of the 
fourth shutter period/v from the shutter opening of the eighth pixel from the left in Fig. 24, are F12/V. 

15 [0183] The foreground component of the second shutter period/v from the shutter opening of the fifth pixel from the 
left in the frame #n in Fig. 24 is F11/V, and the foreground component of the third shutter period/v from the shutter 
opening of the sixth pixel from the left in Fig. 24 is also F11/V. The foreground component of the fourth shutter period/ 
v from the shutter opening of the seventh pixel from the left in Fig. 24 is F11/v. 

[0184] The foreground component of the third shutter period/v from the shutter opening of the fifth pixel from the left 
20 in the frame #n in Fig. 24 is F1 0/v, and the foreground component of the fourth shutter period/v from the shutter opening 
of the sixth pixel from the left in Fig. 24 is also F10/v. The foreground component of the fourth shutter period/v from 
the shutter opening of the fifth pixel from the left in the frame #n in Fig. 24 is F09/v. 

[0185] Since the object corresponding to the background keeps still, the background component of the first shutter 
period/v from the shutter opening of the sixth pixel from the left in the frame #n in Fig. 24 is B05/v. The background 
25 components of the first and second shutter period/vs from the shutter opening of the seventh pixel from the left in the 
frame #n in Fig. 24 are B06/v. The background components of the first through third shutter period/vs from the shutter 
opening of the eighth pixel from the left in the frame #n in Fig. 24 are B07/v. 

[01 86] In the frame #n in Fig. 24, the sixth through eighth pixels from the left belong to the mixed region of the covered 
background region. 

30 [0187] The ninth through twelfth pixels from the left in the frame #n in Fig. 24 belong to the background region, and 
the pixel values are B08 through B11 , respectively. 

[0188] The first through ninth pixels from the left in the frame #n + 1 in Fig. 24 belong to the foreground region. The 
foreground component of the shutter period/v in the foreground region in the frame #n + 1 is one of F01/V through F1 2/v. 
[01 8Q] Since an assumption may be made that the object corresponding to the foreground is a rigid body and moves 

35 at a constant velocity, and the foreground image moves so as to be displayed at a position four pixels to the right side 
in the following frame, the foreground component of the first shutter period/v from the shutter opening of the ninth pixel 
from the left in the frame #n + 1 in Fig. 24 is F12/V, and the foreground component of the second shutter period/v from 
the shutter opening of the tenth pixel from the left in Fig . 24 is also F1 2/v. The foreground component of the third shutter 
period/v from the shutter opening of the eleventh pixel from the left in Fig. 24, and the foreground component of the 

40 fourth shutter period/v from the shutter opening of the twelfth pixel from the left in Fig. 24, are F12/v. 

[0190] The foreground component of the second shutter period/v from the shutter opening of the ninth pixel from the 
left in the frame #n + 1 in Fig, 24 is F11/v, and the foreground component of the third shutter period/v from the shutter 
opening of the tenth pixel from the left in Fig. 24 is also F11/v. The foreground component of the fourth shutter period/ 
v from the shutter opening of the eleventh pixel from the left in Fig. 24 is F11/v. 

45 [0191] The foreground component of the third shutter period/v from the shutter opening of the ninth pixel from the 
left in the frame #n + 1 in Fig. 24 is F1 0/v, and the foreground component of the fourth shutter period/v from the shutter 
opening of the tenth pixel from the left in Fig. 24 is also F1 0/v. The foreground component of the fourth shutter period/ 
v from the shutter opening of the ninth pixel from the left in the frame #n + 1 in Fig. 24 is F09/v. 
[0192] Since the object corresponding to the background keeps still, the background component of the first shutter 

so period/v from the shutter opening of the tenth pixel from the left in the frame #n + 1 in Fig. 24 is B09/v. The background 
components of the first and second shutter period/vs from the shutter opening of the eleventh pixel from the left in the 
frame #n + 1 in Fig. 24 are B10/v. The background components of the first through third shutter period/vs from the 
shutter opening of the twelfth pixel from the left in the frame #n + 1 in Fig. 24 are B11/y. 

[0193] In the frame #n + 1 in Fig. 24, the tenth through twelfth pixels from the left side correspond to the mixed region 
55 which is the covered background region. 

[0194] Fig. 25 is a model diagram wherein the foreground components are extracted from the pixel values illustrated 
in Fig. 24. 

[0195] Fig. 26 is a model diagram wherein the pixel values of the pixels adjacently arrayed in a row in three frames 
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of the images which are taken of the foreground corresponding to the object which moves to the right side in the drawing 
with the still background, and are at the same position in the frames, develop over the time direction. In Fig. 26, the 
model diagram includes the uncovered background region. 

[0196] In Fig. 26, an assumption may be made that the object corresponding to the foreground is a rigid body, and 
5 moves at a constant velocity. Since the object corresponding to the foreground moves so as to be displayed at a position 
four pixels to the right side in the following frame, the movement amount v is 4. 

[01 97] For example, the foreground component of the first shutter period/v from the shutter opening of the left-most 
pixel in the frame #n - 1 in Fig. 26 is F13/v, and the foreground component of the second shutter period/v from the 
shutter opening of the second pixel from the left in Fig. 26 is also F1 3/v. The foreground component of the third shutter 

10 period/v from the shutter opening of the third pixel from the left in Fig. 26, and the foreground component of the fourth 
shutter period/v from the shutter opening of the fourth pixel from the left in Fig. 26, are F13/V. 
[0198] The foreground component of the first shutter period/v from the shutter opening of the second pixel from the 
left in the frame #n - 1 in Fig. 26 is F1 4/v, and the foreground component of the second shutter period/v from the shutter 
opening of the third pixel from the left in Fig. 26 is also F14/v. The foreground component of the first shutter period/v 

15 from the shutter opening of the third pixel from the left in Fig. 26 is F15/v. 

[0199] Since the object corresponding to the background keeps still, the background component of the second 
through fourth shutter period/vs from the shutter opening of the left-most pixel in the frame #n - 1 in Fig. 26 is B25/v. 
The background components of the third and fourth shutter period/vs from the shutter opening of the second pixel from 
the left in the frame #n - 1 in Fig. 26 are B26/v. The background component of the fourth shutter period/v from the 

20 shutter opening of the third pixel from the left in the frame #n - 1 in Fig. 26 is B27/v. 

[0200] In the frame #n - 1 in Fig. 26, the left-most pixel through the third pixel belong to a mixed region of the uncovered 
background region. 

[0201] The fourth through twelfth pixels from the left in the frame #n - 1 in Fig. 26 belong to the foreground region. 
The foreground component in the frame is one of F1 3/v through F24/v. 
25 [0202] The left-most pixel through the fourth pixel from the left in the frame #n in Fig. 26 belong to the background 
region, and the pixel values are B25 through B28, respectively. 

[0203] Since an assumption may be made that the object corresponding to the foreground is a rigid body and moves 
at a constant velocity, and the foreground image moves so as to be displayed at a position four pixels to the right side 
in the following frame, the foreground component of the first shutter period/v from the shutter opening of the fifth pixel 

30 from the left in the frame #n in Fig. 26 is F13/v, and the foreground component of the second shutter period/v from the 
shutter opening of the sixth pixel from the left in Fig. 26 is also F13/v. The foreground component of the third shutter 
period/v from the shutter opening of the seventh pixel from the left in Fig. 26, and the foreground component of the 
fourth shutter period/v from the shutter opening of the eighth pixel from the left in Fig. 26, are F13/V. 
[0204] The foreground component of the first shutter period/v from the shutter opening of the sixth pixel from the left 

35 in the' frame #n in Fig. 26 is F14/V, and the foreground component of the second shutter period/v from the shutter 
opening of the seventh pixel from the left in Fig. 26 is also F1 4/v. The foreground component of the first shutter period/ 
v from the shutter opening of the eighth pixel from the left in Fig. 26 is F1 5/v. 

[0205] Since the object corresponding to the background keeps still, the background components of the second 
through fourth shutter period/vs from the shutter opening of the fifth pixel from the left in the frame #n in Fig. 26 are 
40 B29/V. The background components of the third and fourth shutter period/vs from the shutter opening of the sixth pixel 
from the left in the frame #n in Fig. 26 are B30/v. The background component of the fourth shutter period/v from the 
shutter opening of the seventh pixel from the left in the frame #n in Fig. 26 is B31/v. 

[0206] In the frame #n in Fig. 26, the fifth through seventh pixels from the left belong to the mixed region of the 
uncovered background region. 
45 [0207] The eighth through twelfth pixels from the left in the frame #n in Fig. 26 belong to the foreground region. The 
value corresponding to the period of the shutter period/v in the foreground region in the frame #n is one of F1 3/v through 
F20/V. 

[0208] The left-most pixel through the eighth pixel from the left in the frame #n + 1 in Fig. 26, belong to the background 
region, and the pixel values thereof are B25 through B32, respectively. 

so [0209] Since an assumption may be made that the object corresponding to the foreground is a rigid body and moves 
at a constant velocity, and the foreground image moves so as to be displayed at a position four pixels to the right side 
in the following frame, the foreground component of the first shutter period/v from the shutter opening of the ninth pixel 
from the left in the frame #n + 1 in Fig. 26 is F13/v, and the foreground component of the second shutter period/v from 
the shutter opening of the tenth pixel from the left in Fig. 26 is also F1 3/v. The foreground component of the third shutter 

55 period/v from the shutter opening of the eleventh pixel from the left in Fig. 26, and the foreground component of the 
fourth shutter period/v from the shutter opening of the twelfth pixel from the left in Fig. 26, are F13/v. 
[0210] The foreground component of the first shutter period/v from the shutter opening of the tenth pixel from the 
left in the frame #n + 1 in Fig. 26 is F14/v, and the foreground component of the second shutter period/v from the shutter 
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opening of the eleventh pixel from the left in Fig. 26 Is also F1 4/v. The foreground component of the first shutter period/ 
v from the shutter opening of the twelfth pixel from the left in Fig. 26 is F1 5/v. 

[0211] Since the object corresponding to the background keeps still, the background components of the second 
through fourth shutter period/vs from the shutter opening of the ninth pixel from the left in the frame #n + 1 in Fig. 26 
s are B33/v. The background components of the third and fourth shutter period/vs from the shutter opening of the tenth 
pixel from the left in the frame #n + 1 in Fig. 26 are B34/v. The background component of the fourth shutter period/v 
from the shutter opening of the eleventh pixel from the left in the frame #n + 1 in Fig. 26 is B35/v. 
[021 2] In the frame #n + 1 in Fig. 26, the ninth through eleventh pixels from the left belong to the mixed region of the 
uncovered background region. 

10 [0213] The twelfth pixel from the left in the frame #n + 1 in Fig. 26 belongs to the foreground region. The foreground 
component of the shutter period/v in the foreground region in the frame #n + 1 is one of F13/v through F16/V. 
[0214] Fig. 27 is a model diagram of the image wherein the foreground components are extracted from the pixel 
values shown in Fig. 26. 

[0215] Fig. 28 is a diagram which illustrates how the input image divided into the image in the foreground region, the 
15 image in the background region, the foreground component image in the covered background region, the background 
component image in the covered background region, the foreground component image in the uncovered background 
region, and the background component image in the uncovered background region, correspond to a model diagram 
wherein the pixel values of pixels develop over the time direction. 

[0216] As shown in Fig. 28, the input image is classified into the foreground region, background region, covered 
20 background region, and uncovered background region, by the region specifying unit 1 03. The input image is separated 
into the image in the foreground region, the image in the background region, the foreground component image in the 
covered background region, the background component image in the covered background region, the foreground 
component image in the uncovered background region, the background component image in the uncovered back- 
ground region, by the foreground/background separation unit 105 based upon the specified regions, i.e., the foreground 
25 region, background region, covered background region, and uncovered background region, and the mixture ratio a 
detected by the mixture ratio calculation unit 104. 

[0217] The separated images, i.e., the image in the foreground region, the image in the background region, the 
foreground component image in the covered background region, the background component image in the covered 
background region, the foreground component image in the uncovered background region, the background component 

30 image in the uncovered background region, are processed, respectively. 

[0218] Fig. 29 is a diagram which illustrates an example of the image divided into the foreground region, the back- 
ground region, and the mixed region. The region specifying unit 103 specifies the foreground region, background 
region, and mixed region, of the input image. The image processing device can divide the input image into the image 
of the foreground region, image of the background region, and image of the mixed region, based upon the region 

35 information indicating the foreground region, background region, and mixed region. 

[0219] As shown in Fig. 30, the foreground/background separation unit 1 05 separates the image of the mixed region 
into the foreground component image and the background component image based upon the region information sup- 
plied from the region specifying unit 103 and the mixture ratio a supplied from the mixture ratio calculating unit 104. 
[0220] Fig. 31 illustrates the flowchart which describes the processing for an image with the image processing device 

40 according to the present invention. 

[0221] In Step S101, the region specifying unit 103 specifies the foreground region, background region, covered 
background region, and uncovered background region of the input image, based upon the movement vector and the 
position information thereof supplied from the movement detecting unit 1 02 and the input image. Details of the process- 
ing for region specifying will be described later. 

45 [0222] In Step S102, the mixture ratio calculating unit 104 calculates the mixture ratio a based upon the region 
information supplied from the region specifying unit 103 and the input image. Details of the processing of the mixture 
ratio calculating unit 104 calculating the mixture ratio a will be described later. 

[0223] In Step S1 03, the foreground/background separation unit 1 05 separates the input image into the image in the 
foreground region, the image in the background region, the foreground component image in the covered background 

50 region, the background component image in the covered background region, the foreground component image in the 
uncovered background, and the background component image in the uncovered background region, based upon the 
region information supplied from the region specifying unit 1 03 and the mixture ratio a supplied from the mixture ratio 
calculation unit 104. Details of the processing for separation of an image by the foreground/background separation 
unit 105 will be described later. 

55 [0224] In Step S 1 04, the separated image processing unit 1 06 performs image processing for each of the separated 
images, i.e., the image in the foreground region, the image in the background region, the foreground component image 
in the covered background region, the background component image in the covered background region, the foreground 
component image in the uncovered background region, and the background component image in the uncovered back- 
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ground region, and the processing ends. Details of the image processing performed by the separated image processing 
unit 106 will be described later. 

[0225] As described above, the image processing device according to the present invention separates the input 
image into the image in the foreground region, the image in the background region, the foreground component image 

5 in the covered background region, the background component image in the covered background region, the foreground 
component image in the uncovered background region, and the background component image in the uncovered back- 
ground region, and performs the image processing for each of the separated images, i.e., the image in the foreground 
region, the image in the background region, the foreground component image in the covered background region, the 
background component image in the covered background region, the foreground component image in the uncovered 

10 background region, and the background component image in the uncovered background region, which are separated. 
[0226] Fig. 32 is a block diagram which illustrates an example of the configuration of the region specifying unit 1 03. 
The region specifying unit 1 03, of which configuration is shown in Fig. 32, does not use the movement vectors. Frame 
memory 201 stores the input images in increments of one frame. In the event that the object of the processing is the 
frame #n, the frame memory 201 stores the frame #n • 2 which is two frames previous from the frame #n, the frame 

15 # n - 1 which is one frame previous from the frame #n, the frame #n, the frame #n + 1 which is one frame following the 
frame #n, and the frame #n + 2 which is two frames following the frame #n. 

[0227] A still/motion judgment unit 202-1 reads out the pixel value of the pixel in the frame #n + 2, which is at the 
same position as the position of the pixel on the image, which is the object of specifying the region in the frame #n, 
and the pixel value of the pixel in the frame #n + 1 , which is at the same position as the position of the pixel on the 

20 image, which is the object of specifying the region of the frame #n, from the frame memory 201, and calculates the 
absolute value of the difference between the read out pixel values. The still/motion judgment unit 202-1 judges whether 
or not the absolute value of the difference between the pixel value in the frame #n + 2 and the pixel value in the frame 
#n + 1 is greater than the predetermined threshold value Th, and in the event that judgment is made that the absolute 
value of the difference is greater than the threshold value Th, the still/motion judgment unit 202-1 supplies the still/ 

25 motion j udgment, indicating motion , to a region judgment unit 203-1 . In the event that judgment is made that the absolute 
value of the difference between the pixel value of the pixel in the frame #n.+ 2 and the pixel value of the pixel in the 
frame #n + 1 is equal to or less than the threshold value Th, the still/motion judgment unit 202-1 supplies the still/motion 
judgment, indicating "still", to the region judgment unit 203-1 . 

[0228] A still/motion judgment unit 202-2 reads out the pixel value of the pixel in the frame #n + 1 , which is at the 

30 same position as the position of the pixel on the image, which is the object of specifying the region in the frame #n, 
and the pixel value of pixel which is the object in the frame #n from the frame memory 201 , and calculates the absolute 
value t>f the difference between the pixel values. The still/motion judgment unit 202-2 judges whether or not the absolute 
value of the difference between the pixel value In the frame #n + 1 and the pixel value in the frame #n is greater than 
the predetermined threshold value Th, and in the event that judgment is made that the absolute value of the difference 

35 between the pixel values is greater than the threshold value Th, the still/motion judgment indicating motion is supplied 
to the region judgment unit 203-1 and the region judgment unit 203-2. In the event that judgment is made that the 
absolute value of the difference between the pixel value of the pixel in the frame #n + 1 and the pixel value of the pixel 
in the frame #n is equal to or smaller than the threshold value Th, the still/motion judgment unit 202-2 supplies the still/ 
motion judgment, indicating "still", to the region judgment unit 203-1 and the region judgment unit 203-2. 

40 [0229] The still/motion judgment unit 202-3 reads out the pixel value of the pixel, which is the object of specifying 
the region in the frame #n, and the pixel value of the pixel in the frame #n - 1 , which is at the same position as the 
position on the image of the pixel, which is the object of specifying the region in the frame #n, from the frame memory 
201 , and calculates the absolute value of the difference between the pixel values. The still/motion judgment unit 202-3 
judges whether or not the absolute value of the difference between the pixel value in the frame #n and the pixel value 

45 in the frame #n - 1 is greater than the predetermined value Th, and in the event that judgment is made that the absolute 
value of the difference between the pixel values is greater than the threshold value Th, the still/motion judgment indi- 
cating motion is supplied to the region judgment unit 203-2 and the region judgment unit 203-3. In the event that 
judgment is made that the absolute value of the difference between the pixel value of the pixel in the frame #n and the 
pixel value of the pixel in the frame #n - 1 is equal to or smaller than the threshold value Th, the still/motion judgment 

so unit 202-3 supplies the still/motion judgment indicating "still" to the region judgment unit 203-2 and the region judgment 
unit 203-3. 

[0230] The still/motion judgment unit 202-4 reads out the pixel value of the pixel in the frame #n - 1 at the same 
position as the position of the pixel on the image, which is the object of specifying the region in the frame #n, and the 
pixel value of the pixel in the frame #n - 2 at the same position as the position of the pixel on the image, which is the 
55 object of specifying the region in the frame #n, from the frame memory 201 , and calculates the absolute value of the 
difference between the pixel values. The still/motion judgment unit 202-4 judges whether or not the absolute value of 
the difference between the pixel value in the frame #n - 1 and the pixel value in the frame #n - 2 is greater than the 
predetermined threshold value Th, and in the event that judgment is made that the absolute value of the difference 
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between the pixel values is greater than the threshold value Th, the still/motion judgment indicating motion is supplied 
to the region judgment unit 203-3. In the event that judgment is made that the absolute value of the difference between 
the pixel value of the pixel in the frame #n - 1 and the pixel value of the pixel in the frame #n - 2 is equal to or smaller 
than the threshold value Th, the still/motion judgment unit 202-4 supplies the still/motion judgment indicating "still" to 
the region judgment unit 203-3. 

[0231] In the event that the still/motion judgment supplied from the still/motion judgment unit 202-1 indicates "still", 
and the still/motion judgment supplied from the still/motion judgment unit 202-2 indicates motion, the region judgment 
unit 203-1 judges that the pixel which is the object of specifying the region in the frame #n belongs to the uncovered 
background region, and sets the uncovered background region judgment flag corresponding to the judged pixel in the 
region, to "1", which indicates that the pixel belongs to the uncovered background region. 

[0232] In the event that the still/motion judgment supplied from the still/motion judgment unit 202-1 indicates motion, 
or the still/motion judgment supplied from the still/motion judgment unit 202-2 indicates still, the region judgment unit 
203-1 judges that the pixel which is the object of specifying the region in the frame #n does not belong to the uncovered 
background region, and sets the uncovered background region judgment flag corresponding to the judged pixel in the 
region to "0", which indicates that the pixel does not belong to the uncovered background region. 
[0233] The region judgment unit 203-1 supplies the uncovered background region judgment flag which has been set 
to "1 " or "0", as described above, to the judgment flag storing memory 204. 

[0234] In the event that the stili/motion judgment supplied from the still/motion judgment unit 202-2 indicates "still", 
and the still/motion judgment supplied from the still/motion judgment unit 202-3 indicates "still", the region judgment 
unit 203-2 judges that the pixel which is the object of specifying the region in the frame #n belongs to the still region, 
and sets the still region judgment flag corresponding to the pixel judged in the region, to "1 ", which indicates that the 
pixel belongs to the still region. 

[0235] In the event that the still/motion judgment supplied from the still/motion judgment unit 202-2 indicates motion, 
or the still/motion judgment supplied from the still/motion judgment unit 202-3 indicates motion, the region judgment 
unit 203-2 judges that the pixel which is the object of specifying the region in the frame #n does not belong to the still 
region, and sets the still region judgment flag corresponding to the judged pixel in the region, to "0", which indicates 
that the pixel does not belong to the still region. 

[0236] The region judgment unit 203-2 supplies the still region judgment flag which has been set to "1" or "0" as 
described above, to judgment flag storing frame memory 204. 

[0237] In the event that the still/motion judgment supplied from the still/motion judgment unit 202-2 indicates motion, 
and the still/motion judgment supplied from the still/motion judgment unit 202-3 indicates motion, the region judgment 
unit 203-2 judges the pixel which is the object of specifying the region in the frame #n belongs to the moving region, 
and sets the moving region judgment flag corresponding to the judged pixel in the region, to "1", which indicates that 
the pixel belongs to the moving region. 

[0238] In the event that the still/motion judgment supplied from the still/motion judgment unit 202-2 indicates "still", 
or the still/motion judgment supplied from the still/motion judgment unit 202-3 indicates "still", the region judgment unit 
203-2 judges that the pixel which is the object of specifying the region in the frame #n does not belong to the moving 
region, and sets the moving region judgment flag corresponding to the judged pixel in the region, to "0", which indicates 
that the pixel does not belong to the moving region. 

[0239] The region judgment unit 203-2 supplies the moving region judgment flag which has been set to "1" or "0", 
to the judgment flag storing frame memory 204. 

[0240] In the event that the still/motion judgment supplied from the still/motion judgment unit 202-3 indicates motion, 
and the still/motion judgment supplied from the stili/motion judgment unit 202-4 indicates "still", the region judgment 
unit 203-3 judges that the pixel which is the object of specifying the region in the frame #n belongs to the covered 
background region, and sets the covered background region judgment flag corresponding to the judged pixel in the 
region to "1", which indicates that the pixel belongs to the covered background region. 

[0241] In the event that the still/motion judgment supplied from the still/motion judgment unit 202-3 indicates "still", 
or the still/motion judgment supplied from the still/motion judgment unit 202-4 Indicates motion, the region judgment 
unit 203-3 judges that the pixel which is the object of specifying the region in the frame #n does not belong to the 
covered background region, and sets the covered background region judgment flag corresponding to the judged pixel 
in the region to "0", which indicates that the pixel does not belong to the covered background region. 
[0242] The region judgment unit 203-3 supplies the covered background region judgment flag which has been set 
to "1" or "0" as described above, to the judgment flag storing frame memory 204. 

[0243] The judgment flag storing frame memory 204 stores the uncovered background region judgment flag supplied 
from the region judgment unit 203-1, the still region judgment flag supplied from the region judgment unit 203-2, the 
moving region judgment flag supplied from the region judgment unit 203-2, and the covered background region judg- 
ment flag supplied from the region judgment unit 203-3. 

[0244] The judgment flag storing frame memory 204 supplies the uncovered background region judgment flag, the 
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still region judgment flag, the moving region judgment flag, and the covered background region judgment flag, which 
are stored therein, to a synthesizing unit 205. The synthesizing unit 205 generates the region information which indi- 
cates which of the uncovered background region, the still region, the moving region, or the covered background region, 
each pixel belongs to, and supplies the information to judgment flag storing frame memory 206, based upon the un- 
covered background region judgment flag, the still region judgment flag, the moving region judgment flag, and the 
covered background region judgment flag, which are supplied from the judgment flag storing frame memory 204. 
[0245] The judgment flag storing frame memory 206 stores the region information supplied from the synthesizing 
unit 205, and also outputs the stored region information. 

[0246] An example for processing performed by the region specifying unit 1 03 will now be described with reference 
to Fig. 33 through Fig. 37. 

[0247] In the event that the object corresponding to the foreground moves, the position of the image corresponding 
to the object on the screen changes with each frame. As shown in Fig. 33, in the frame #n, the image corresponding 
to the object which is at the position indicated by Yn(x,y) is at the position Yn + 1(x,y) in the following frame #n + 1 . 
[0248] Fig. 34 is a model diagram wherein the pixel values of pixels of the image corresponding to the foreground 
object, which are adjacently arrayed in sequence in a image movement direction, develop over the time direction. For 
example, in the event that the image moving direction corresponding to the foreground object is horizontal to the screen, 
the model diagram in Fig. 34 indicates the model wherein the pixel values of adjacent pixels in one line develop over 
the time direction. 

[0249] In Fig. 34, the line in the frame #n is the same as the line in the frame #n + 1 . 

[0250] The foreground components corresponding to the object, which are included in the second pixel through 
thirteenth pixel from the left in the frame #n, are included in the sixth through seventeenth pixels from the left in the 
frame #n + 1 , 

[0251] In the frame #n, the pixels belonging to the covered background region are the eleventh through thirteenth 
pixels from the left, and the pixels belonging to the uncovered background region are the second through fourth pixels 
from the left. In the frame #n + 1, the pixels belonging to the covered background region are the fifteenth through 
seventeenth pixels from the left, and the pixels belonging to the uncovered background region are sixth through eighth 
pixels from the left. 

[0252] With the example shown in Fig. 34, the movement amount v is 4, since the foreground components included 
in the frame #n move by four pixels in the frame #n + 1 . The virtual dividing number is 4, corresponding to the movement 
value v. 

[0253] Next, a description will be made regarding the change of the pixel values of the pixels belonging to the mixed 
region in the frames previous to and following the frame of interest. 

[0254] . In the frame #n wherein the background keeps still and the movement amount v of the foreground is 4, shown 
in Fig. 35, the pixels belonging to the covered background region are the fifteenth through seventeenth pixels from the 
left. Since the movement amount v is 4, in the previous frame #n - 1 , the fifteenth through seventeenth pixels from the 
left include only the background components, and belong to the background region. Also, in the frame # n - 2 which 
is one further before, the fifteenth through seventeenth pixels from the left contain only the background components, 
and belong to the background region. 

[0255] Note that since the object corresponding to the background keeps still, the pixel value of the fifteenth pixel 
from the left in the frame #n - 1 do not change from the pixel value of the fifteenth pixel from the left in the frame #n - 
2. Similarly, the pixel value of the sixteenth pixel from the left in the frame #n - 1 do not change from the pixel value of 
the sixteenth pixel from the left in the frame #n - 2, and the pixel values of the seventeenth pixel from the left in the 
frame #n - 1 do not change from the pixel value of the seventeenth pixel from the left in the frame #n - 2. 
[0256] That is to say, the pixels of the frame #n - 1 and frame #n - 2 corresponding to the pixels belonging to the 
covered background region in the frame #n consists of only the background components, and the pixel values do not 
change, and accordingly the absolute value of the difference therebetween is approximately zero. Accordingly, judg- 
ment is made that the still/motion judgment for the pixels of the frame #n - 1 and the frame #n - 2 corresponding to the 
pixels belonging to the mixed region in the frame #n is still by the still/motion judgment unit 202-4. 
[0257] Since the pixels belonging to the covered background region in the frame #n contain the foreground compo- 
nents, the pixel values are different from the case wherein the pixel values in the frame #n - 1 consist of only the 
background components. Accordingly, judgment is made that the still/motion judgment for the pixels belonging to the 
mixed region in the frame #n and the pixels in the frame #n - 1 corresponding thereto is motion by the still/motion 
judgment unit 202-3. 

[0258] As described above, the region judgment unit 203-3 judges that the corresponding pixels belong to the covered 
background region in the event that the still/motion judgment unit 202-3 supplies the results of the still/motion judgment 
which indicates motion, and the still/motion judgment unit 202-4 supplies the results of the still/motion judgment which 
indicates "still". 

[0259] In the frame #n wherein the background keeps still and the foreground movement amount v is 4 as shown in 
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Fig. 36, the pixels included in the uncovered background region are the second through fourth pixels from the left. 
Since the movement amount v is 4, in the frame #n + 1 following the frame #n, the second through fourth pixels from 
the left include only the background components, and belong to the background region. Also, in the frame #n + 2 further 
one frame following the frame #n + 1, the second through fourth pixels from the left contain only the background 

5 components, and belong to the background region. 

[0260] Note that since the object corresponding to the background keeps still, the pixel values of the second pixel 
from the left in the frame #n + 2 does not change from the pixel value of the second pixel from the left in the frame #n 
+ 1 . Similarly, the pixel value of the third pixel from the left in the frame #n + 2 does not change from the pixel value of 
the third pixel from the left in the frame #n + 1 , and the pixel value of the fourth pixel from the left in the frame #n + 2 

w does not change from the pixel value of the fourth pixel from the left in the frame #n + 1 . 

[0261] That is to say, the pixels of the frame #n + 1 and the frame #n + 2, corresponding to the pixels belonging to 
the uncovered background region in the frame #n, consist of only the background components, so the pixel values 
thereof do not change, and accordingly the absolute value of the difference thereof is approximately zero. Accordingly, 
judgment is made that the still/motion judgment for the pixels of the frame #n + 1 and the frame #n + 2 corresponding 

15 to the pixels belonging to the mixed region in the frame #n is "still" by the still/motion judgment unit 202-1 . 

[0262] Since the pixels belonging to the uncovered background region in the frame #n contain the foreground com- 
ponents, the pixel values are different from the case wherein the pixels consists of only the background components 
in the frame #n + 1 . Accordingly, judgment is made that the still/motion judgment for the pixels belonging to the mixed 
region in the frame #n and the pixels corresponding thereto in the frame #n + 1 is motion by the still/motion judgment 

20 unit 202-2. 

[0263] As described above, the region judgment unit 203-1 judges that the corresponding pixels belong to the un- 
covered background region in the event that the still/motion judgment unit 202-2 supplies the results of the still/motion 
judgment which indicates motion, and the still/motion judgment unit 202-1 supplies the still/motion judgment which 
indicates -still". 

25 [0264] Fig. 37 is a diagram which illustrates judgment conditions of the region specifying unit 1 03 in the frame #n. 
In the event that judgment is made that the pixel in the frame #n - 2 at the same position as the position of the pixel 
which is the object of judgment on the image in the frame #n, and the pixel in the frame #n - 1 at the same position as 
the position of the pixel which is the object of judgment on the image in the frame #n, are "still", and judgment is made 
that the pixel in the frame #n - 1 at the same position as the position otthe pixel which is the object of judgment on the 

30 image in the frame #n, and the pixel in the frame #n are motion, the region specifying unit 103 judges that the pixel 
which is the object of judgment of the frame #n belongs to the covered background region. 

[0265]" In the event that judgment is made that the pixel In the frame #n - 1 at the same position as the position of 
the pixej which is the object of judgment on the image in the frame #n, and the pixel In the frame #n, are judged to be 
"still", and judgment is made that the pixel in the frame #n and the pixel in the frame #n + 1 at the same position as 

35 the position of the pixel which is the object of judgment on the image in the frame #n, are judged to be "still", the region 
specifying unit 1 03 judges that the pixel which is the object of judgment of the frame #n belongs to the still region. 
[0266] In the event that judgment is made that the pixel in the frame #n - 1 at the same position as the position of 
the pixel which is the object of judgment on the image in the frame #n, and the pixel in the frame #n, are judged to be 
motion, and judgment is made that the pixel of the frame #n and the pixel in the frame #n + 1 at the same position as 

^o the position of the pixel which is the object of judgment on the image in the frame #n, are judged to be motion, the 
region specifying unit 1 03 judges that the pixel which is the object of judgment of the frame #n belongs to the movement 
region. 

[0267] In the event that judgment is made that the pixel of the frame #n and the pixel in the frame #n + 1 at the same 
position as the position of the pixel which is the object of judgment on the image in the frame #n, are motion, and 

45 judgment is made that the pixel in the frame #n + 1 at the same position as the position of the pixel which is the object 
of judgment on the image in the frame #n, and the pixel in the frame #n + 2 at the same position as the position of the 
pixel which is the object of judgment on the image in the frame #n, are judged to be "still", the region specifying unit 
1 03 judges that the pixel which is the object of judgment of the frame #n belongs to the uncovered background region. 
[0268] Fig. 38A through Fig. 38D are diagrams which illustrate examples of results of the region specifying unit 1 03 

50 specifying the region. In Fig. 38A, the pixels which have been judged to belong to the covered background region are 
displayed in white. In Fig. 38B, the pixels which have been judged to belong to the uncovered background region are 
displayed in white. 

[0269] In Fig. 38C, the pixels which have been judged to belong to the movement region are displayed in white. In 
Fig. 38D, the pixels which have been judged to belong to the still region are displayed in white. 
55 [0270] Fig; 39 is a diagram which illustrates the region information as an image, indicating the mixed region of the 
region information which the judgment flag storing frame memory 206 outputs. In Fig. 39, the pixels which have been 
judged to belong to the covered background region or the uncovered background region, i.e., the pixels judged to 
belong to the mixed region, are displayed in white. The region information indicating the mixed region, which the judg- 
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ment flag storing frame memory 206 outputs, indicates the mixed region and the portions which have texture within 
the foreground region and are surrounded by portions which have no texture. 

[0271] Next, referring to the flowchart in Fig. 40, the processing for region specifying by the region specifying unit 
103 will be described. In Step S201 , the frame memory 201 obtains the images of the frame #n - 2 through the frame 

5 #n + 2, including the frame #n which is the object of judgment. 

[0272] In Step S202, the still/motion judgment unit 202-3 judges whether or not the pixel of the frame #n - 1 and the 
pixel of the frame #n at the same position keep still, and in the event of judgment of "still 0 , the flow proceeds to Step 
S203, and the still/motion judgment unit 202-2 judges whether or not the pixel of the frame #n and the pixel of the 
frame #n + 1 at the same position keep still. 

w [0273] In Step S203, in the event that judgment is made that the pixel of the frame #n and the pixel of the frame #n 
+ 1 at the same position are "still", the flow proceeds to Step S204, and the region judgment unit 203-2 sets the still 
region judgment flag corresponding to the judged pixel in the region to "1" which indicates the pixel belongs to the still 
region. The region judgment unit 203-2 supplies the still region judgment flag to the judgment flag storing frame memory 
204, and the procedure proceeds to Step S205. 

15 [0274] In Step S202, in the event that judgment is made that the pixel of the frame #n - 1 and the pixel of the frame 
#n at the same position are motion, or in Step S203, judgment is made that the pixel of the frame #n and the pixel of 
the frame #n + 1 at the same position are motion, the pixel of the frame #n does not belong to the still region, and 
accordingly the processing in Step S204 is skipped, and the procedure proceeds to Step S205. 
[0275] In Step S205, the still/motion judgment unit 202-3 judges whether or not the pixel of the frame #n - 1 and the 

20 pixel of the frame #n at the same position are in motion, and in the event of judgment of motion, the flow proceeds to 
Step S206, and the still/motion judgment unit 202-2 judges whether or not the pixel of the frame #n and the pixel of 
the frame #n + 1 at the same position are in motion. 

[0276] In Step S206, in the event that judgment is made that the pixel of the frame #n and the pixel of the frame #n 
+ 1 at the same position are in motion, the flow proceeds to Step S207, the region judgment unit 203-2 set the movement 
25 region judgment flag corresponding to the judged pixel in the region to "1" which indicates that the pixel belongs to the 
movement region. The region judgment unit 203-2 supplies the movement region judgment flag to the judgment flag 
storing frame memory 204, and the procedure proceeds to Step S208. 

[0277] In Step S205, in the event that judgment is made that the pixel of the frame #n - 1 and the pixel of the frame 
#n at the same position are "still", or in Step S206, in the event that judgment is made that the pixel of the frame #n 

30 and the pixel of the frame #n + 1 at the same position are "still", since the pixel of the frame #n does not belong to the 
movement region, the processing in Step S207 is skipped, and the procedure proceeds to Step S208. 
[0278] In Step S208, the still/motion judgment unit 202-4 judges whether or not the pixel of the frame #n - 2 and the 
pixel of the frame #n - 1 at the same position keeps still, and in the event of judgment of "still", the flow proceeds to 
Step S209, and the still/motion judgment unit 202-3 judges whether or not the pixel of the frame #n - 1 and the pixel 

35 of the frame #n at the same position are in motion. 

[0279] In Step S209, in the event that judgment is made that the pixel of the frame #n - 1 and the pixel of the frame 
#n at the same position are in motion, the flow proceeds to Step S210, and the region judgment unit 203-3 sets the 
covered background region judgment flag corresponding to the judged pixel in the region to "1" which indicates that 
the pixel belongs to the covered background region. The region judgment unit 203-3 supplies the covered background 

^0 region judgment flag to the judgment flag storing frame memory 204, and the procedure proceeds to Step S211 . 

[0280] In Step S208, in the event that judgment is made that the pixel of the frame #n -2 and the pixel of the frame 
#n - 1 at the same position are in motion, or in Step S209, in the event that judgment is made that the pixel of the frame 
#n - 1 and the pixel of the frame #n at the same position are "still", the pixel of the frame #n does not belong to the 
covered background region, so the processing in Step S210 is skipped, and the procedure proceeds to Step S211. 

45 [0281] In Step S211 , the still/motion judgment unit 202-2 judges whether or not the pixel of the frame #n and the 
pixel of the frame #n + 1 at the same position are in motion, and in the event of judgment of motion, the flow proceeds 
to Step S212, and the stiil/motion judgment unit 202-1 judges whether or not the pixel of the frame #n + 1 and the pixel 
of the frame #n + 2 at the same position keep still. 

[0282] In Step S212, in the event that judgment is made that the pixel of the frame #n + 1 and the pixel of the frame 
50 #n + 2 at the same position are "still", the flow proceeds to Step S213, and the region judgment unit 203-1 sets the 
uncovered background region judgment flag corresponding to the judged pixel in the region to "1" which indicates that 
the pixel belongs to the uncovered background region. The region judgment unit 203-1 supplies the uncovered back- 
ground region judgment flag to the judgment flag storing frame memory 204, and the procedure proceeds to Step S21 4. 
[0283] In Step S21 1 , in the event that judgment is made that the pixel of the frame #n and the pixel of the frame #n 
55 + 1 at the same position are "still", or in Step 212, in the event that judgment is made that the pixel of the frame #n + 
1 and the pixel of the frame #n + 2 at the same position are in motion, since the pixel of the frame #n does not belong 
to the uncovered background region, the processing in Step S21 3 is skipped, and the procedure proceeds to Step S21 4. 
[0284] In Step S21 4, the region specifying unit 103 judges whether or not all the pixels in the frame #n are region- 
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specified, and in the event that judgment is made that not all pixels are region-specified, the procedure returns to Step 
S202, and repeats the processing of specifying the region for other pixels. 

[0285] In Step S214, in the event that judgment is made that all the pixels in the frame #n are region-specified, the 
flow proceeds to Step S215, and the synthesizing unit 205 generates the region information which indicates the mixed 
5 region based upon the uncovered background region judgment flag and the covered background region judgment flag, 
which are stored in the judgment flag storing frame memory 204, and furthermore generates the region information 
which indicates which of the uncovered background region, the still region, the movement region, or the covered back- 
ground region , each pixel belongs to, sets the generated region inf ormation for the judgment flag storing frame memory 
206, and the processing ends. 

10 [0286] As described above, the region specifying unit 1 03 can generate region information which indicates which of 
the movement region, the still region, the uncovered background region, or the covered background region, each pixel 
contained in the frame belongs to. 

[0287] Note that an arrangement may be made wherein the region specifying unit 103 generates the region infor- 
mation corresponding to the mixed region and the region information made up of flags which indicates which of the 
is movement region, the still region, or the mixed region, each of pixels contained in the frame belongs to, by applying 
the logical sum to the region information corresponding to the uncovered background region and the covered back- 
ground region. 

[0288] In the event that the object corresponding to the foreground has texture, the region specifying unit 1 03 can 
specify the movement region more accurately. 
20 [0289] The region specifying unit 1 03 can output the region information indicating the movement region as the region 
information indicating the foreground region, or output the region information indicating the still region as the region 
information indicating the background region. 

[0290] While description has been made wherein the object corresponding to the background keeps still, the process- 
ing of specifying the region described above can be applied even if the image corresponding to the background region 

25 contains motion. For example, in the event that the image corresponding to the background region moves in a constant 
manner, the region specifying unit 1 03 shifts the entire image corresponding to the movement, and performs processing 
in the same manner as with the case wherein the object corresponding to the background keeps still. Also, in the event 
that the image corresponding to the background region contains a different motion at each local position, the region 
specifying unit 103 selects the pixel corresponding to the motion, and performs the above-described processing. 

30 [0291] Fig. 41 is a block diagram which illustrates another example of the structure of the region specifying unit 1 03. 
The region specifying unit 1 03 shown in Fig. 41 does not use movement vectors. A background image generating unit 
301 generates the background image corresponding to the input image, and supplies the generated background image 
to a binary object image extracting unit 302. The background image generating unit 301 extracts, for example, the 
image object corresponding to the background object contained in the input image, and generates the background 

35 image. 

[0292] An example of a model diagram is illustrated in Fig. 42 wherein the pixel values of the pixels arrayed in 
sequence adjacently in a movement direction of the image corresponding to the foreground object develop over the 
time direction. For example, the model diagram in Fig. 42 illustrates a model wherein, in the event that the movement 
direction of the image corresponding to the foreground object is horizontal to the screen, the pixel values of the adjacent 

40 pixels in one line develop over the.time direction. 

[0293] In Fig. 42, the line in the frame #n is the same as the line in the frame #n - 1 and the line in the frame #n + 1 . 
[0294] In the frame #n, the foreground components corresponding to the object, which are contained in the sixth 
pixel through seventeenth pixel from the left, are contained in the second through thirteenth pixels from the left in the 
frame #n - 1 , and are contained in the tenth through twenty first pixels from the left in the frame #n + 1 . 

45 [0295] In the frame #n - 1 , the pixels belonging to the covered background region are the eleventh through thirteenth 
pixels from the left, and the pixels belonging to the uncovered background region are the second through fourth pixels 
from the left. In the frame #n, the pixels belonging to the covered background region are the fifteenth through the 
seventeenth pixels from the left, and the pixels belonging to the uncovered background region are the sixth through 
eighth pixels from the left. In the frame #n + 1 , the pixels belonging to the covered background region are the nineteenth 

so through twenty first pixels from the left, and the pixels belonging to the uncovered background region are the tenth 
through twelfth pixels from the left. 

[0296] In the frame #n - 1 , the pixels belonging to the background region are the first from the left, and the fourteenth 
through twenty first pixels from the left. In the frame #n, the pixels belonging to the background region are the first 
through fifth pixels from the left, and the eighteenth through twenty first pixels from the left. In the frame #n + 1 , the 
55 pixels belonging to the background region are the first through ninth pixels from the left. 

[0297] An example of the background image corresponding to the example shown in Fig. 42, which is generated by 
the background image generating unit 301 , is illustrated in Fig. 43. The background image is made up of the pixels 
corresponding to the background object, and does not contain image components corresponding to the foreground 
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object. 

[0298] The binary object image extracting unit 302 generates a binary object image based upon the correlation 
between the background image and the input image, and supplies the generated binary object image to a time change 
detecting unit 303. 

5 [0299] Fig. 44 is a block diagram which illustrates the configuration of the binary object image extracting unit 302. A 
correlation value computing unit 321 computes the correlation between the background image supplied from the back- 
ground image generating unit 301 and the input image, generates a correlation value, and supplies the generated 
correlation value to a threshold value processing unit 322. 

[0300] The correlation value computing unit 321 applies Expression (4) to a block 3x3 wherein X 4 is centered in 
10 the background image as shown in Fig. 45A, and a block 3x3 wherein Y 4 corresponding to the block in the background 
image is centered in the input image as shown in Fig. 45B, and calculates a correlation value corresponding to the Y 4 , 
for example. 

t{Xi-X)£(Yi-Y) 
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[0301] The correlation value computing unit 321 supplies the correlation value calculated corresponding to each 
35 pixel as described above to the threshold value processing unit 322. 

[0302] Also, an arrangement may be made wherein the correlation value computing unit 321 , for example, applies 
Expression (7) to the block 3 x 3 in the background image wherein X 4 is centered as shown in Fig. 46A, and the block 
3 x 3 in the input image wherein Y 4 is centered corresponding to the block in the background image as shown in Fig. 
46B, and calculates the sum of absolute value of difference corresponding to Y 4 . 



Sum of Absolute Value = yj j . — Yi)\ (7) 
of Difference i=o 



[0303] The correlation value computing unit 321 supplies the difference absolute value calculated as described above 
as the correlation value to the threshold value processing unit 322. 

[0304] The threshold value processing unit 322 compares the pixel value of the correlation image with the threshold 
value thO, and in the event that the correlation value is equal to or less than the threshold value thO, the threshold value 
so processing unit 322 sets the pixel value of the binary object image to 1 , and in the event that the correlation value is 
greater than the threshold value thO, the threshold value processing unit 322 sets the pixel value of the binary object 
image to 0, and outputs the binary object image of which each pixel value has been set to 0 or 1 . The threshold value 
processing unit 322 may store the threshold value thO beforehand, and may use the threshold value thO which is input 
externally. 

55 [0305] Fig. 47 is a diagram which illustrates an example of the binary object image corresponding to the model of 
the input image shown in Fig. 42. In the binary object image, a pixel value of a pixel having a high correlation with the 
background image is set to 0. 

[0306] Fig. 48 is a block diagram which illustrates the configuration of the time change detecting unit 303. Frame 
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memory 341 stores the binary object images of the frame #n - 1 , frame #n, and frame #n + 1 , supplied from the binary 
object image extracting unit 302 at the point of judgment of the region for the pixel of the frame #n. 
[0307] A region judgment unit 342 judges the region for each pixel of the frame #n based upon the binary object 
images of the frame #n -1 , frame #n p and frame #n + 1 , which are stored in the frame memory 341 , generates the 
5 region information, and outputs the generated region information. 

[0308] Fig. 49 is a diagram which describes the judgment made by the region judgment unit 342. In the event that 
the pixel of interest of the binary object image of the frame #n is 0, the region judgment unit 342 judges the pixel of 
interest of the frame #n to belong to the background region. 

[0309] In the event that the pixel of interest of the binary object image of the frame #n is 1 , the corresponding pixel 
10 of the binary object image of the frame #n - 1 is 1 , and the corresponding pixel of the binary object image of the frame 

#n + 1 is 1 , the region judgment unit 342 judges the pixel of interest of the frame #n to belong to the foreground region. 

[0310] In the event that the pixel of interest of the binary object image of the frame #n is 1 , and the corresponding 

pixel of the binary object image of the frame #n - 1 is 0, the region judgment unit 342 judges the pixel of interest of the 

frame #n to belong to the covered background region. 
*5 [0311] In the event that the pixel of interest of the binary object image of the frame #n is 1 , and the corresponding 

pixel of the binary object image of the frame #n + 1 is 0, the region judgment unit 342 judges the pixel of interest of 

the frame #n to belong to the uncovered background region. 

[031 2] Fig. 50 is a diagram which illustrates an example wherein the time change detecting unit 303 judges the binary 
object image corresponding to the mode! of the input image shown in Fig. 42. The time change detecting unit 303 
20 judges the first through fifth pixels from the left of the frame #n to belong to the background region since the corre- 
sponding pixels of the binary object image of the frame #n are 0. 

[031 3] The time change detecting unit 303 judges the sixth through ninth pixels from the left to belong to the uncovered 
background region since the pixels of the binary object image of the frame #n are 1 , and the corresponding pixels of 
the frame #n + 1 are 0. 

25 [0314] The time change detecting unit 303 judges the tenth through thirteenth pixels from the left to belong to the 
foreground region since the pixels of the binary object image of the frame #n are 1, the corresponding pixels of the 
frame #n - 1 are 1 , and the corresponding pixels of the frame #n + 1 are 1 . 

[031 5] The time change detecting unit 303 judges the fourteenth through seventeenth pixels from the left to belong 
to the covered background region since the pixels of the binary object image of the frame #n are 1 , and the corre- 

30 sponding pixels of the frame #n - 1 are 0. 

[0316].. The time change detecting unit 303 judges the eighteenth through twenty first pixels from the left to belong 
to the background region since the corresponding pixels of the binary object image of the frame #n are 0. 
[0317] *fThe processing of specifying the region by the region judgment unit 103 will be now described, referring to 
the flowchart shown in Fig. 51 . In Step S301 , the background image generating unit 301 of the region judgment unit 

35 103, for example, generates the background image by extracting the image object corresponding to the background 
object contained in the input image based upon the input image, and supplies the generated background image to the 
binary object image extracting unit 302. 

[0318] In Step S302, the binary object image extracting unit 302 computes the correlation value between the input 
image and the background image supplied from the background image generating unit 301 by the computation de- 
40 scribed referring to Fig. 45, for example. In Step S303, the binary object image extracting unit 302 computes the binary 
object image from the correlation value and the threshold value thO by comparing the correlation value with the threshold 
value thO, for example. 

[031 9] In Step S304, the time change detecting unit 303 performs processing of region judgment, and the processing 
ends. 

45 [0320] The processing of the region judgment corresponding to Step S304 will be described in detail, referring to the 
flowchart shown in Fig. 52. In Step S321 , the region judgment unit 342 of the time change detecting unit 303 judges 
whether or not the pixel of interest in the frame #n stored in the frame memory 341 is 0, and in the event that the 
judgment is made that the pixel of the interest in the frame #n is 0, the flow proceeds to Step S322, makes settings to 
the effect that the pixel of interest in the frame #n belongs to the background region, and the processing ends. 

so [0321 ] I n Step S32 1 , in the event that judgment is made that the pixel of interest in the frame #n is 1 , the flow proceeds 
to Step S323, and the region judgment unit 342 of the time change detecting unit 303 judges whether or not the pixel 
of interest in the frame #n stored in the frame memory 341 is 1 , and the corresponding pixel in the frame #n - 1 is 0, 
and in the event that judgment is made that the pixel of interest in the frame #n is 1 , and the corresponding pixel in the 
frame #n - 1 is 0, the flow proceeds to Step S324, makes settings to the effect that the pixel of interest in the frame #n 

55 belongs to the covered background region, and the processing ends. 

[0322] In Step S323, in the event that judgment is made that the pixel of interest in the frame #n is 0, or the corre- 
sponding pixel in the frame #n - 1 is 1 , the flow proceeds to Step S325, and the region judgment unit 342 of the time 
change detecting unit 303 judges whether or not the pixel of interest in the frame #n stored in the frame memory 341 
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is 1 , and the corresponding pixel in the frame #n + 1 is 0, and in the event that judgment is made that the pixel of 
interest in the frame #n is 1 , and the corresponding pixel in the frame #n + 1 is 0, the flow proceeds to Step S326, 
makes settings to the effect that the pixel of interest in the frame #n belongs to the uncovered background region, and 
the processing ends. 

s [0323] In Step S325, in the event that judgment is made that the pixel of interest in the frame #n is 0, or the corre- 
sponding pixel in the frame #n + 1 is 1 , the flow proceeds to Step S327, and the region judgment unit 342 of the time 
change detecting unit 303 sets the pixel of interest in the frame #n for the foreground region, and the processing ends. 
[0324] As described above, the region specifying unit 1 03 can specify which of the foreground region, the background 
region, the covered background region, or the uncovered background region, the pixel of the input image belongs to, 

10 and can generate region information corresponding to the specified results. 

[0325] Fig. 53 is a block diagram which illustrates another configuration of the region specifying unit 1 03. The region 
specifying unit 103 shown in Fig. 53 uses the movement vector and the position information thereof, which are supplied 
from the movement detecting unit 1 02. Portions the same as those shown in Fig. 41 are denoted by the same reference 
numerals, and description thereof will be omitted. 

15 [0326] A robustification unit 361 generates a robustified binary object image based upon N frames of the binary 
object image supplied from the binary object image extracting unit 302, and outputs to the time change detecting unit 
303. 

[0327] Fig. 54 is a block diagram which describes the configuration of the robustification unit 361 . A movement 
compensation unit 381 compensates for the movement of the binary object image of N frames based upon the move- 
20 ment vector and the position information thereof supplied from the movement detecting unit 1 02, and outputs the binary 
object image which has been subjected to compensation of movement to a switch 382. 

[0328] The movement compensation of the movement compensation unit 381 will be described with reference to 
examples shown in Fig. 55 and Fig. 56. For example, when judging the region in the frame #n, in the event that there 
is input of the binary object images of the frame #n - 1, the frame #n, and the frame #n + 1, shown by way of the 
25 example in Fig. 55, the movement compensation unit 381 compensates for movement of the binary object image of 
the frame #n - 1 and the binary object image of the frame #n + 1 , based upon the movement vector supplied from the 
movement detecting unit 102, and supplies the binary object image which has been subjected to compensation of 
movement to the switch 382, as indicated in the example shown in Fig. 56. 

[0329] The switch 382 outputs the binary object image which has been subjected to movement compensation of the 
30 first frame, to the frame memory 383-1 , and outputs the binary object image which has been subjected to movement 
compensation of the second frame to the frame memory 383-2. 

[0330f Similarly, the switch 382 outputs each of the binary object images of which the third through N - 1'th frames 
have begn subjected to compensation for the movement to each of frame memory 383-3 through frame memory 383-(N 
- 1), respectively, and outputs the binary object image of which the Nth frame has been subjected to movement com- 

35 pensstion to frame memory 383- N. 

[0331] The frame memory 383-1 stores the binary object image of which the first frame has been subjected to move- 
ment compensation, and outputs the stored binary object image to a weighting addition unit 384-1 . The frame memory 
383-2 stores the binary object image of which the second frame has been subjected to movement compensation, and 
outputs the stored binary object image to a weighting addition unit 384-2. 

40 [0332] Similarly, each of the frame memory 383-3 through the frame memory 383-(N - 1 ) stores each of the binary 
object images of which one of the third frame through N • 1th frame has been subjected to compensation for the 
movement, and outputs the stored binary object image to each of the weighing addition unit 384-3 through the weighing 
addition unit 384-(N - 1). The frame memory 383-N stores the binary object image of which Nth frame has been 
subjected to compensation for the movement, and outputs the stored binary object image to a weighing addition unit 

« 384-N. 

[0333] The weighing addition unit 384-1 multiplies the pixel value of the binary object image of which the first frame 
has been subjected to compensation for the movement supplied from the frame memory 383-1 by the predetermined 
weight w1 , and supplies to an accumulation unit 385. The weighing addition unit 384-2 multiplies the pixel value of the 
binary object image of the second frame which has been subjected to movement compensation supplied from the 

50 frame memory 383-2 by the predetermined weight w2, and supplies to an accumulation unit 385. 

[0334] Similarly, each of the weighting addition unit 384-3 through the weighing addition unit 384-(N - 1 ) multiplies 
the pixel value of the binary object image of one of the third through N - Vth frames, which has been subjected to 
movement compensation supplied from one of the frame memory 383-3 through the frame memory 383-(N - 1 ) by one 
of the predetermined weights w3 through w(N - 1), and supplies to the accumulation unit 385. A weighing addition unit 

55 384-N multiplies the pixel value of the binary object image of the N'th frame supplied from the frame memory 383 - N 
which has been subjected to movement compensation by the predetermined weight wN, and supplies to the accumu- 
lation unit 385. 

[0335] The accumulation unit 385 accumulates the pixel value corresponding to the binary object image, wherein 
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each of the first through Nth frames which has been subjected to movement compensation is multiplied by one of the 
predetermined weights w1 through wN, and generates the binary object image by comparing the accumulated pixel 
value with the predetermined threshold value thO. 

[0336] As described above, the robustification unit 361 generates the robustified binary object image from the N 
5 frames of binary object images, and supplies to the time change detecting unit 303, so the region specifying unit 1 03 
of which the configuration is shown in Fig. 53 can specify the region more accurately as compared with the case shown 
in Fig. 41 , even if the input image contains noise. 

[0337] The processing for specifying the region of the region specifying unit 1 03 of which the configuration is shown 
in Fig. 53 will now be described, referring to the flowchart shown in Fig. 57. The processing in Step S341 through Step 
10 S343 is the same as Step S301 through Step S303 described in the flowchart shown in Fig. 51, respectively, and 
accordingly, description thereof will be omitted. 

[0338] In Step S344, the robustification unit 361 performs processing for robustification. 

[0339] In Step S345, the time change detecting unit 303 performs processing for specifying the region, and the 
processing ends. Details of the processing in Step S345 are the same as the processing described with reference to 

15 the flowchart shown in Fig. 52, so description thereof will be omitted. 

[0340] Referring to the flowchart shown in Fig. 58, processing of robustification corresponding to the processing in 
Step S344 shown in Fig. 57 will now be described in detail. In Step S361, the movement compensation unit 381 
performs movement compensation processing of the input binary object image based upon the movement vector and 
the position information thereof supplied from the movement detecting unit 1 02. In Step S362, one of the frame memory 

20 383-1 through the frame memory 383-N stores the binary object image, which has been subjected to movement com- 
pensation, supplied via the switch 362. 

[0341] In Step S363, the robustification unit 361 judges whether or not N binary object images are stored, and in the 
event that judgment is made that N binary object images have not been stored, the flow returns to Step S361 , and the 
robustification unit 363 repeats processing of compensation for movement of the binary object image, and processing 
25 of storing the binary object image. 

[0342] In Step S363, in the event that judgment is made that N binary object images stored, the flow proceeds to 
Step S364, and each of the weighting addition units 384-1 through 384-N multiplies each of N binary object images, 
by one of the weights w1 through wN for weighting. 

[0343] In Step S365, the accumulation unit 385 accumulates the N weighted binary object images. 
30 [0344] In Step S366, the accumulation unit 385 generates the binary object image from the accumulated image, by 
comparing with the predetermined threshold value th1 , for example, and the processing ends. 
[0345? As described above, the region specifying unit 1 03, of which the configuration is shown in Fig. 53, can generate 
region information based upon the robustified binary object image. 

[0346] * As described above, the region specifying unit 1 03 can generate the region information which indicates which 
35 of the movement region, the still region, the uncovered background region, or the covered background region, each 
of the pixels contained in the frame belongs to. 

[0347] Fig. 59 is a block diagram which illustrates an example of the configuration of the mixture ratio calculating 
unit 1 04. An estimated mixture ratio processing unit 401 calculates estimated mixture ratio for each pixel by computation 
corresponding to a model of a covered background region based upon the input image, and supplies the calculated 
40 estimated mixture ratio to a mixture ratio determination unit 403. 

[0348] An estimated mixture ratio processing unit 402 calculates estimated mixture ratio for each pixel by computation 
corresponding to a model of the uncovered background region based upon the input image, and supplies the calculated 
estimated mixture ratio to the mixture ratio determination unit 403. 

[0349] Since an assumption may be made that the object corresponding to the foreground moves at a constant 
45 velocity within a shutter period, the mixture ratio a of a pixel belonging to the mixed region has a nature such as 
described below. That is to say, the mixture ratio a changes linearly corresponding to the change of the position of the 
pixel. Taking the change of the pixel position to be one-dimensional, the change of the mixture ratio a may be repre- 
sented by a straight line, and taking the change of the pixel position to be two-dimensional, the change of the mixture 
ratio a may be represented by a plane. 
50 [0350] Note that the period of one frame is short, an assumption may be made that the object corresponding to the 
foreground is a rigid body, and moves at a constant velocity. 

[0351] In this case, the inclination of the mixture ratio a is inversely proportionate to the movement amount v of the 
foreground within the shutter period. 

[0352] An example of an ideal mixture ratio a is shown in Fig. 60. The inclination 1 of an ideal mixture ratio a in the 
55 mixed region may be represented by the reciprocal of the movement amount v. 

[0353] As shown in Fig. 60, an ideal mixture ratio a has a value of 1 in the background region, and has a value of 0 

in the foreground region, and has a value which exceeds 0 and is less than 1 in the mixed region. 

[0354] With the example shown in Fig, 61 , the pixel value C06 of the seventh pixel from the left in the frame #n may 
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be represented in Expression (8), using the pixel value P06 of the seventh pixel from the left in the frame #n - 1 . 

C06 = B06lv + B06Iv + F01/v + F02lv 
= P06lv + P06/v + FOllv + F02/v 
=2JvP06+£Fi/v (8) 

10 

[0355] In Expression (8), the pixel value C06 is represented as the pixel value M of the pixel in the mixed region, 
and the pixel value P06 is represented as the pixel value B of the pixel in the background region. That is to say, the 
pixel value M of the pixel in the mixed region and the pixel value B of the pixel in the background region may be 
represented as in Expression (9) and Expression (10), respectively. 

15 

M = C06 (9) 



20 B = P06 (10) 

[0356] In Expression (8), 2/v corresponds to the mixture ratio a. Since the movement amount v is 4, the mixture ratio 
a of the seventh pixel from the left in the frame #n is 0.5. 

[0357] As described above, Expression (3) indicating the mixture ratio a may be rewritten as with Expression (11) 
25 by reckoning the pixel value C in the frame #n of interest to be a pixel value in the mixed region, and reckoning the 
pixel value P in the frame #n - 1 previous to the frame #n to be a pixel value of the background region. 

C = a-P + f (11) 

30 

[0358] In Expression (11), f is the sum of the foreground components contained in the pixel of interest, L,Fi/v. The 
variables included in Expression (11) are two, i.e., the mixture ratio a and the sum of the foreground components f. 
[0359]^ 3 In the same way, Fig. 62 illustrates a model wherein the pixel values wherein the movement amount v is 4, 
and virtual dividing number is 4 in the uncovered background region, develop over the time direction. 
35 [0360] Expression (3) indicating the mixture ratio a may be represented as in Expression (1 2) with the pixel value C 
in the frame #n of interest as a pixel value in the mixed region, and with the pixel value N in the frame #n + 1 following 
the frame #n as a pixel value in the background region, in the same manner as the representation in the covered 
background region described above, in the uncovered background region. 

40 C = a-N + f (12) 

[0361] Note that while description has been made with an assumption that the background object keeps still, Ex- 
pression (8) through Expression (1 2) may be applied by using the pixel values of the pixels at the positions correspond- 
45 ing to the background movement amount v, even if the background object moves. For example, in Fig. 61 , in the event 
that the movement amount v of the object corresponding to the background is 2, the virtual dividing number is 2, and 
the object corresponding to the background moves to the right side in the drawing, the pixel value B of the pixel in the 
background region in Expression (10) is the pixel value P04. 

[0362] Expression (11 ) and Expression (12) include two variables, respectively, and accordingly the mixture ratio a 
so can not be obtained in this state. Here, images generally have great correlation spatially, the adjacent pixels have 
approximately the same value. 

[0363] Thus, since the foreground components have great correlation spatially, the mixture ratio is obtained by trans- 
forming the expression so as to obtain the sum of the foreground components from the previous or following frame. 
[0364] The pixel value Mc of the seventh pixel from the left in the frame #n in Fig. 63 may be represented in Expression 

55 (13). 
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Mc = 4- m B06 +%Filv (13) 

V Ml 

[0365] The first argument 2/v of the right side in Expression (13) corresponds to the mixture ratio a. The second 
argument of the right side in Expression (1 3) is represented as in Expression (1 4) using the pixel values in the following 
frame #n + 1 . 



12 10 

£Fi/v = /3.£Fi7v (14) 

i=JJ . i = 7 

[0366] Here, an assumption may be made that Expression (1 5) holds, using the spatial correlation of the foreground 
components. 

F = F05 = F06 = F07 = F08 = F09 = F10 = F1 1 = F12 (15) 
[0367] Expression (14) may be rewritten as Expression (16) using Expression (15). 

i=u v 

[0366] As a result, p may be represented in Expression (17). 

p = 2/4 (17) 

[0369] In general, in the event that an assumption is made wherein the foreground components correlated to the 
mixed region are the same as shown in Expression (15), Expression (18) is formed by the relationship of the internal 
dividing ratio for all the pixel in the mixed region. 

P = T-a (18) 
[0370] In the event that Expression (18) holds, Expression (11) may develop as indicated in Expression (19). 

C = a*P+f 

r+v-j 

= a*P+(l-a)*2,Fi/v 

= a>P+(i-ot)-N < 19 > 

[0371] Similarly, in the event that Expression (18) holds, Expression (12) may develop as indicated in Expression (20). 
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r+v-v 

=a-N+(l-a)-p < 2 °) 

10 [0372] In Expression (19) and Expression (20), since C, N, and P are known pixel values, the variable included in 
Expression (19) and Expression (20) is only the mixture ratio a. The relationship between C, N, and P in Expression 
(1 9) and Expression (20) is illustrated in Fig. 64. C is the pixel value of the pixel of interest in the frame #n for calculating 
the mixture ratio a. N is the pixel value of the pixel in the frame #n + 1 , of which the position in the spatial direction 
corresponds to that of the pixel of interest. P is the pixel value of the pixel in the frame #n - 1 , of which the position in 

15 the spatial direction corresponds to that of the pixel of interest. 

[0373] Accordingly, since Expression (1 9) and Expression (20) include one variable each, the mixture ratio a can be 
calculated using the pixel values in three frames. The conditions for calculating an accurate mixture ratio a by solving 
Expression (19) and Expression (20) are that; the foreground components with regard to the mixed region are the 
same, that is to say, in the foreground image object which has been taken in the state of the foreground object being 

20 still, the pixel values of pixels of a number double the movement amount v, which are arrayed sequentially at the 
boundary of the image object, corresponding to the movement direction of the foreground object, are constant. 
[0374] As described above, the mixture ratio a of the pixel belonging to the covered background region is calculated 
by Expression (21 ), and the mixture ratio a belonging to the uncovered background region is calculated by Expression 
(22), 

25 

a = (C-N)/(P-N) (21) 



30 a = (C-P)/(N-P) (22) 

[0375] Fig. 65 is a block diagram which illustrates the configuration of the estimated mixture ratio processing unit 
401. Ffcame memory 421 stores the input image in increments of frames, and supplies the frame following the frame 
which is input as an input image, to frame memory 422 and a mixture ratio computation unit 423. 
35 [0376] The frame memory 422 stores the input image in increments of frames, and supplies the frame following the 
frame supplied from the frame memory 421 , to the mixture ratio computation unit 423. 

[0377] Accordingly, in the event that the frame #n + 1 is input as an input image to the mixture ratio computation unit 
423, the frame memory 421 supplies the frame #n to the mixture ratio computation unit 423, and the frame memory 
422 supplies the frame #n - 1 to the mixture ratio computation unit 423. 

40 [0378] The mixture ratio computation unit 423 calculates the estimated mixture ratio of the pixel of interest by the 
computation represented in Expression (21) based upon the pixel value C of the pixel of interest in the frame #n, the 
pixel value N of the pixel in the frame #n + 1 wherein the spatial position thereof corresponds to that of the pixel of 
interest, and the pixel value P of the pixel in the frame #n - 1 wherein the spatial position thereof corresponds to that 
of the pixel of interest, and outputs the calculated estimated mixture ratio. For example, in the event that the background 

45 keeps still, the mixture ratio computation unit 423 calculates the estimated mixture ratio of the pixel of interest based 
upon the pixel value C of the pixel of interest in the frame #n, the pixel value N of the pixel-in the frame #n + 1 at the 
same position in the frame as the pixel of interest, and the pixel value P of the pixel in the frame #n - 1 at the same 
position in the frame as the pixel of interest, and outputs the calculated estimated mixture ratio. 
[0379] As described above, the estimated mixture ratio processing unit 401 can calculate the estimated mixture ratio 

50 based upon the input image, and supply to the mixture ratio decision unit 403. 

[0380] Note that the processing of the estimated mixture ratio processing unit 402 is the same as that of the estimated 
mixture ratio processing unit 401 except for the processing wherein, while the estimated mixture ratio processing unit 
401 calculates the estimated mixture ratio of the pixel of interest by the computation represented in Expression (21), 
the estimated mixture ratio processing unit 402 calculates the estimated mixture ratio of the pixel of interest by the 

55 computation represented in Expression (22), and accordingly, description thereof will be omitted. 

[0381] Fig. 66 is a diagram which illustrates an example of the estimated mixture ratio calculated by the estimated 
mixture ratio processing unit 401 . The estimated mixture ratio shown in Fig. 66 indicates the results in a case wherein 
the foreground movement amount v corresponding to the object which moves at a constant velocity is 11 , for one line. 
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[0382] It can be understood that the estimated mixture ratio changes generally linearly in the mixed region, as shown 
in Fig. 60. 

[0383] Returning to Fig. 59, the mixture ratio decision unit 403 sets the mixture ratio a based upon the region infor- 
mation indicating which of the foreground region, the background region, the covered background region, or the un- 
s covered background region, the pixel which is the object of calculation of the mixture ratio a belongs to, supplied from 
the region specifying unit 103. 

[0384] In the event that the pixel which is the object belongs to the foreground region, the mixture ratio decision unit 
403 sets the mixture ratio a to 0, in the event that the pixel which is the object belongs to the background region, sets 
the mixture ratio a to 1 , in the event that the pixel which is the object belongs to the covered background region, sets 
10 the mixture ratio a to the estimated mixture ratio supplied from the estimated mixture ratio processing unit 401 , and in 
the event that the pixel which is the object belongs to the uncovered background region, sets the mixture ratio a to the 
estimated mixture ratio supplied from the estimated mixture ratio processing unit 402. The mixture ratio decision unit 
403 outputs the mixture ratio a which has been set based upon the region information. 

[0385] Fig. 67 is a block diagram which illustrates another configuration of the mixture ratio calculating unit 104. A 
15 selection unit 441 supplies the pixels belonging to the covered background region and the corresponding pixels in the 
following and previous frames, to an estimated mixture ratio processing unit 442, based upon the region information 
supplied from the region specifying unit 103. The selection unit 441 supplies the pixels belonging to the uncovered 
background region and the corresponding pixels in the previous and following frames, to an estimated mixture ratio 
processing unit 443, based upon the region information supplied from the region specifying unit 103. 
20 [0386] The estimated mixture ratio processing unit 442 calculates the estimated mixture ratio of the pixel of interest 
belonging to the covered background region by the computation represented in Expression (21) based upon the pixel 
values input from the selection unit 441 , and supplies the calculated estimated mixture ratio to a selection unit 444. 
[0387] The estimated mixture ratio processing unit 443 calculates the estimated mixture ratio of the pixel of interest 
belonging to the uncovered background region by the computation represented in Expression (22) based upon the 
25 pixel values input from the selection unit 441 , and supplies the calculated estimated mixture ratio to the selection unit 
444. 

[0388] In the event that the pixel which is the object belongs to the foreground region, the selection unit 444 selects 
the estimated mixture ratio of 0, and sets for the mixture ratio a, and in the event that the pixel which is the object 
belongs to the background region, the selection unit 444 selects the estimated mixture ratio of 1, and sets for the 

30 mixture ratio a, based upon the region information supplied from the region specifying unit 103. In the event that the 
pixel which is the object belongs to the covered background region, the selection unit 444 selects the estimated mixture 
ratio supplied from the estimated mixture ratio processing unit 442, and sets for the mixture ratio a, and in the event 
that pixel which is the object belongs to the uncovered background region, the selection unit 444 selects the esti- 
mated mixture ratio supplied from the estimated mixture ratio processing unit 443, and sets this for the mixture ratio 

35 a. The selection unit 444 outputs the mixture ratio a which has been selected and set based upon the region information. 
[0389] As described above, the mixture ratio calculating unit 104 having another configuration shown in Fig. 67 can 
calculate the mixture ratio a for each pixel contained in the image, and output the calculated mixture ratio a. 
[0390] Referring to the flowchart shown in Fig. 68, the processing for calculation of the mixture ratio a by the mixture 
ratio calculating unit 104 of which configuration is shown in Fig. 59 will be described. In Step S401, the mixture ratio 

40 calculating unit 104 obtains the region information supplied from the region specifying unit 103. In Step S402, the 
estimated mixture ratio processing unit 401 performs processing of computation of the estimated mixture ratio by a 
model corresponding to the covered background region, and supplies the calculated estimated mixture ratio to the 
mixture ratio decision unit 403. Details of the processing for computation of the estimated mixture ratio will be described 
later with reference to the flowchart shown in Fig. 69. 

45 [0391] In Step S403, the estimated mixture ratio processing unit 402 performs the processing of the computation of 
the estimated mixture ratio by a model corresponding to the uncovered background region, and supplies the calculated 
mixture ratio to the mixture ratio decision unit 403. 

[0392] In Step S404, the mixture ratio calculating unit 1 04 judges whether or not the mixture ratio a has been esti- 
mated for the entire frame, and in the event that judgment is made that the mixture ratio a has not been estimated for 
so the entire frame, the flow returns to Step S402, and performs the processing of estimation of the mixture ratio a for 
the following pixel. 

[0393] In the event that judgment is made in Step S404 that the mixture ratio a has been estimated for the entire 
frame, the flow proceeds to Step S405, and the mixture ratio decision unit 403 sets the mixture ratio a based upon the 
region information which indicates which of the foreground region, the background region, the covered background 
55 region, or the uncovered background region, the pixel belongs to, supplied from the region specifying unit 103. In the 
event that the pixel which is the object belongs to the foreground region, the mixture ratio decision unit 403 sets the 
mixture ratio a to O, in the event that the pixel which is the object belongs to the background region, sets the mixture 
ratio a to 1 , in the event that the pixel which is the object belongs to the covered background region, sets the mixture 
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ratio a to the estimated mixture ratio supplied from the estimated mixture ratio processing unit 401, and in the event 
that the pixel which is the object belongs to the uncovered background region, sets the mixture ratio a to the estimated 
mixture ratio supplied from the estimated mixture ratio processing unit 402, and the processing ends. 
[0394] As described above, the mixture ratio calculating unit 1 04 can calculate the mixture ratio a which is the amount 
5 of features corresponding to each pixel based upon the region information supplied from the region specifying unit 1 03 
and the input image. 

[0395] The processing of calculation of the mixture ratio a by the mixture ratio calculation unit 1 04 of which config- 
uration is shown in Fig. 67 is the same as the processing described in the flowchart shown in Fig. 68, so description 
thereof will be omitted. 

w [0396] The processing for mixture ratio estimation by a mode! corresponding to the covered background region, 
which corresponds to Step S402 in Fig. 68, will now be described with reference to the flowchart shown in Fig. 69. 
[0397] In Step S421 , the mixture ratio computation unit 423 obtains the pixel value C of the pixel of interest in the 
frame #n from the frame memory 421 . 

[0398] In Step S422, the mixture ratio computation unit 423 obtains the pixel value P of the pixel in the frame #n - 
15 1 ( which corresponds to the pixel of interest, from the frame memory 422. 

[0399] In Step S423, the mixture ratio computation unit 423 obtains the pixel value N of the pixel in the frame #n + 
1 , which corresponds to the pixel of interest contained in the input image. 

[0400] In Step S424, the mixture ratio computation unit 423 computes the estimated mixture ratio based upon the 
pixel value C of the pixel of interest in the frame #n, the pixel value P of the pixel in the frame #n - 1, and the pixel 

20 value N of the pixel in the frame #n + 1 . 

[0401] In Step S425, the mixture ratio computation unit 423 judges whether or not the processing for computation 
of the estimated mixture ratio has been ended for the entire frame, and in the event that judgment is made that the 
processing for computation of the estimated mixture ratio has not been ended for the entire frame, the flow returns to 
' Step S421 , and the processing for calculating of the estimated mixture ratio is repeated for the following pixel. 

25 [0402] In Step S425, in the event that judgment is made that the processing for computation of the estimated mixture 
ratio has been ended for the entire frame, the processing ends. 

[0403] As described above, the estimated mixture ratio processing unit 401 can compute the estimated mixture ratio 
based upon the input image. 

[0404] The processing of mixture ratio estimation by a model corresponding to the uncovered background region 
30 shown in Step S403 in Fig. 68 is the same as the processing indicated in the flowchart shown in Fig. 69, wherein 
expressions corresponding to a model of the uncovered background region are used, and accordingly description 
thereof will be omitted. 

[0405] Note that the estimated mixture ratio processing unit 442 and the estimated mixture ratio processing unit 443 
shown in Fig. 67 compute the estimated mixture ratio by performing the processing the same as the processing indicated 

35 in the flowchart shown in Fig. 69, and accordingly description thereof will be omitted. 

[0406] Also, while description has been made with an assumption that the object corresponding to the background 
keeps still, the processing for obtaining the mixture ratio a described above may be applied even if the image corre- 
sponding to the background region contains movement. For example, in the event that the image corresponding to the 
background moves uniformly, the estimated mixture ratio processing unit 401 shifts the entire image corresponding to 

^o the background movement, and performs processing in the same manner as in the case wherein the object corre- 
sponding to the background keeps still. Also, in the event that the image corresponding to the background region 
contains the background movement which is different at each local position, the estimated mixture ratio processing 
unit 401 selects the pixels corresponding to the background movement as the pixels corresponding to the pixels be- 
longing to the mixed region, and performs the processing described above. 

45 [0407] Also, an arrangement may be made wherein the mixture ratio calculating unit 104 performs only the processing 
of the mixture ratio estimation by a model corresponding to the covered background region for all pixels, and outputs 
the calculated estimated mixture ratio as the mixture ratio a. In this case, the mixture ratio a indicates the ratio of the 
background components with regard to the pixels belonging to the covered background region, and indicates the ratio 
of the foreground components with regard to the pixels belonging to the uncovered background region. The image 

so processing device can obtain the mixture ratio a indicating the ratio of the background components with regard to the 
pixels belonging to the uncovered background region, by calculating the absolute value of the difference between the 
mixture ratio a calculated as described above and 1 , and setting the calculated absolute value for the mixture ratio a, 
with regard to the pixels belonging to the uncovered background region. 

[0408] Note that similarly, an arrangement may be made wherein the mixture ratio calculating unit 1 04 performs only 
55 the processing for the mixture ratio estimation by a model corresponding to the uncovered background region for all 
pixels, and outputs the calculated estimated mixture ratio as the mixture ratio a. 
[0409] Another processing of the mixture ratio calculation unit 1 04 will now be described. 

[0410] An expression wherein the mixture ratio a and the sum of the foreground components f are approximated 
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spatially can be formed, using the nature wherein the mixture ratio a changes linearly corresponding to the change of 
the pixel position due to the object corresponding to the foreground moving at a constant velocity within a shutter 
period. The mixture ratio a is calculated by solving the expression wherein the mixture ratio a and the sum of the 
foreground components f are approximated, using multiple sets of the pixel value of the pixel belonging to the mixed 
5 region and the pixel value of the pixel belonging to the background region. 

[0411] In the event that the change of the mixture ratio a is generally linearly, the mixture ratio a is represented in 
Expression (23). 

a = il + p (23) 

[0412] In Expression (23), i denotes the index in the spatial direction wherein the position of the pixel of interest is 
0. 1 is the inclination of the straight line of the mixture ratio a. p is the intercept of the straight line of the mixture ratio 
a, as well as the mixture ratio a of the pixel of interest. In Expression (23), while the index i is known, the inclination 1 
15 and the intercept p are unknown. 

[0413] The correlation between the index i, the inclination 1 , and the intercept p, is shown in Fig. 70. 

[0414] In Fig. 70 and Fig. 71 , white circles indicate the pixel of interest. In Fig. 70, solid circles indicate pixels near 

the pixel of interest. 

. [0415] Multiple different mixture ratio a for multiple pixels are represented by two variables by approximating the 
20 mixture ratio a as in Expression (23). In the example shown in Fig. 70, five mixture ratios for five pixels are represented 
by two variables, i.e., the inclination 1 and the intercept p. 

[0416] In the event of approximating the mixture ratio a in a planner manner shown in Fig. 71 , taking the movement 
v corresponding to the two di rections of the horizontal direction and the vertical direction of the image into consideration , 
the mixture ratio a is represented in Expression (24) by expanding Expression (23) onto a plane. 

25 

a = jm + kq + p (24) 

[0417] In Expression (24), j is the index in the horizontal direction wherein the position of the pixel of interest is 0, 
30 and k is the index in the vertical direction, m is the inclination of the mixture ratio a in the horizontal direction, and q is 
the inclination of the plane of the mixture ratio a in the vertical direction, p is the intercept of the plane of the mixture 
ratio a. 

[0418] For example, in the frame #n shown in Fig. 61 , Expression (25) through Expression (27) hold with regard to 
C05 through C07, respectively. 

35 

C05 = ct05 • B05/V + f05 (25) 
C06=ct06-B06/v + f06 (26) 

40 

C07 = a07 • B07/V + f07 (27) 

45 [0419] Making an assumption that the foreground components generally agree, i.e., that F01 through F03 are the 
same, and that F01 through F03 are written as Fc, Expression (28) holds. 

f(x) = (1 - o(x)) • Fc (28) 

50 

[0420] In Expression (28), x denotes the position in the spatial direction. 

[0421] Rewriting a(x) as Expression (24), Expression (28) may be represented as Expression (29). 

55 f(x) = (1-(jm + kq + p))-Fc 

= j.(-m.Fc) + k.(-q.Fc) + ((1 - p) . Fc) 
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= js + kt + u(29) (29) 
[0422] In Expression (29), (-m • Fc), (-q • Fc), and (1 -p) • Fc are rewritten as Expression (30) through Expression (32). 

5 

s = -m • Fc (30) 
10 t = -q-Fc (31) 

u = (1 - p) ■ Fc (32) 

is [0423] In Expression (29), j is the index in the horizontal direction wherein the position of pixel of interest is 0, and 
k is the index in the vertical direction. 

[0424] As described above, since an assumption is made that the object corresponding to the foreground moves at 
a constant velocity within a shutter period, and the components corresponding to the foreground generally agree, the 
sum of the foreground components is approximated in Expression (29). 
20 [0425] Note that in the event of approximating the mixture ratio a linearly, the sum of the foreground components 
may be represented in Expression (33). 

f(x) = is + u (33) 

25 

[0426] Rewriting the mixture ratio a and the sum of the foreground components in Expression (13) using Expression 
(24) and Expression (29), the pixel value M is represented in Expression (34). 

M = (jm + kq + p) • B + js + kt + u 
= jB-m + kB.q + B.p + j.s + k.t + u . (34) 



[0427] In Expression (34), the unknown variables are the six values of the inclination of the plane of the mixture ratio 
35 a in the horizontal direction, m, the inclination of the plane of the mixture ratio a in the vertical direction, q, the intercepts 
of the plane of the mixture ratio a, p, s, t, and u. 

[0428] Setting the pixel value M and pixel value B for the normal equation represented in Expression (34) corre- 
sponding to the pixels near the pixel of interest, the mixture ratio a is calculated by solving multiple normal equations 
wherein the pixel value M and the pixel value B have been set, by the least square method. 
40 [0429] For example, with the index j of the pixel of interest in the horizontal direction as 0, with the index_k of the 
pixel of interest in the vertical direction as 0, and setting the pixel value M or the pixel value B for the normal equation 
represented in Expression (34) with regard to 3 x 3 pixels near the pixel of interest, Expression (35) through Expression 
(43) are obtained. 

45 M. V1 = (-1) • B. V1 • m + (-1) • B. V1 ■ q + B. V1 . p + 

(-1).s + (-1).t + u (35) 

50 ^ = (0) • Bo ., • m + (-1) • B 0r1 • q + B^ ■ p + (0) 

■s + (-1)-t+u (36) 



M +V1 = (+1) • B +V1 • m + (-1) • B +vl • q + B +V1 • p + 

(+1).s + (-1)-t+u (37) 



38 



EP1 396 819 A1 



10 



15 



20 



25 



30 



M.i,o = (-1) • B. 10 • m + (0) • B. 1(0 • q + B. 10 • p + 

(-1)-s + (0)t + u (38) 

M o.o = (°) • B o,o • m + (0) . B 0 o • q + B 0 o • P + (0) • s 

+ (0) . t + u (39) 

M +10 = (+1) • B +10 • m + (0) • B +1|0 • q + B +1 0 . p + 

(+1)-s + (0)-t + u (40) 

M. 1i+1 = (-1) • B. 1+1 • m + (+1) - B_ V1 • q + B. 1+1 - p + 

(-1).s + (+1)-t+'u (41) 

M 0,+1 = (0) ' B 0l+ 1 ' m + (+1) • B 0f+1 • q + B 0 +1 . p + (0) 

.S + (+1)-t + U (42) 

M + 1. + 1 - (+1) • B +1i+1 • m + (4-1) • B +1+1 . q + B +1+1 ■ p + 

(+1).s + (+1).t + u (43) 



[0430] Since the index of the pixel of interest in the horizontal direction, j, is 0, and the index in the vertical direction, 
k, is 0, the mixture ratio a of the pixel of interest is equal to the value wherein j = 0 and k = 0, from Expression (24), i. 
e. t the intercept p. 

[0431] Accordingly, the intercept p can be output as the mixture ratio a by calculating the inclination in the horizontal 
35 direction, m, the inclination in the vertical direction, q, the intercept p, s, t, and u, by the least square method, based 
upon the nine expressions of Expression (35) through Expression (43). 

[0432] More specific procedures for calculating the mixture ratio a by applying the least square method will now be 
described. 

[0433] Representing the index i and the index k with one index x, the relationship between the index i, the index k, 
40 and the index x, is represented in Expression (44). 

x = (i + 1)-3 + (k+1) (44) 

45 [0434] The inclination in the horizontal direction, m, the inclination in the vertical direction, q, the intercept p, s, t, and 
u, are represented by variables, wO, w1 , w2, w3, w4, and w5, respectively, and JB f kB, B, j, k, and I are represented 
by a0, a1 , a2, a3, a4, and a5, respectively. Taking the margin of error ex into consideration, Expression (35) through 
Expression (43) are rewritten as Expression (45). 



50 



Mx = Jl^ay • wy + ex {45) 



55 [0435] In Expression (45), x denotes one of the integers between 0 and 8. 
[0436] Expression (46) may be derived from Expression (45). 
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ex = Mx -i^ay-Wy (46) 

[0437] To apply the least square method, the sum of squares of margin of error E is defined as represented in Ex- 
pression (47). 



10 



E=tex> (47) 

15 

[0438] To minimize the margin of error, the partial derivative of the squared-sum of the margin of error E from the 
variable Wv should be 0. Here v is one of the integers between 0 through 5. Accordingly, wy is calculated so as to 
satisfy Expression (48). 



20 



25 



30 



dwv ~ 2 Ai ex dwx 



[0439] Substituting Expression (46) for Expression (48), Expression (49) is obtained. 



(48) 



s s s 

2(fl r £fl)" Wy) -^av-Mx 

-= 0 yaO x= 0 



(49) 



35 

[0440] Applying, for example, the sweeping method (Gauss-Jordan elimination) and so forth, to the six expressions 
each of which is obtained by substituting one of the integers between 0 and 5 for v in Expression (49), wy is calculated. 
As described above, wO denotes the inclination in the horizontal direction, m, w1 denotes the inclination in the vertical 
direction, q, w2 denotes the intercept p, w3 denotes s, w4 denotes t, and w5 denotes u. 
40 [0441] As described above, the. inclination in the horizontal direction, m, the inclination in the vertical direction, q, 
the intercept p, s, t, and u may be obtained by applying the least square method to an expression wherein the pixel 
value M and the pixel value B have been set. 

[0442] In the description corresponding to Expression (35) through Expression (43), while description has been made 

with the pixel value of the pixel included in the mixed region as M, and the pixel value of the pixel included in the 
45 background region as B, the normal equation needs to be formed for each case of the pixel of interest being included 

in the covered background region, and being included in the uncovered background region. 

[0443] For example, in a case of obtaining the mixture ratio a of the pixel included in the covered background region 

of the frame #n shown in Fig. 61 , the pixels of the frame #n, C04 through C08, and the pixel values of the pixels of the 

frame #n - 1 , P04 through P08, are set for the normal equation. 
so [0444] In a case of obtaining the mixture ratio a of the pixel included in the uncovered background region of the 

frame #n shown in Fig. 62, the pixels of the frame #n, C28 through C32, and the pixel values of the pixels of the frame 

#n + 1 , N28 through N32, are set for the normal equation. 

[0445] Also, for example, in the event of calculating the mixture ratio a of the pixel included in the covered background 
region shown in Fig. 72, Expression (50) through Expression (58) shown below may be formed. The pixel value of the 
55 pixel for calculation of the mixture ratio a is Mc5. 

Mc1 =(-1)>Bc1 -m + (-1)-Bc1 -q + Bc1 -p + (-1)- 
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10 



s + (-1).t + u (50) 

Mc2 = (0) . Bc2 . m + (-1) • Bc2 . q + Bc2 • p + (0) • s 

+ (-1)-t+u (51) 

Mc3 = (+1) • Bc3 • m + (-1) • Bc3 . q + Bc3 . p + (+1) . 

s + (-1)-t + u (52) 

15 Mc4 = (-1) . Bc4 . m + (0) • Bc4 ■ q + Bc4 • p + (-1) • 

s + (0) • t + u (53) 

2Q Mc5 = (0) • Bc5 • m + (0) • Bc5 • q + Bc5 • p + (0) • s 

+ (0)-t + u (54) 



25 



30 



40 



Mc6 = (+1) • Bc6 • m + (0) • Bc6 • q + Bc6 • p + (+1) • 

s + (0) ■ t + u (55) 

Mc7 = (-1) ■ Bc7 • m + (+1) ■ Bc7 • q + Bc7 . p + (-1) ■ 

s + (+1).t + u (56) 

Mc8 = (0) • Bc8 . m + (+1) ■ Bc8 • q + Bc8 - p + (0) • s 

+ (+1).t+u " (57) 

Mc9 = (+1) • Bc9 • m + (+1) • Bc9 • q + Bc9 • p + (+1) ■ 

s + ( + 1). t + u (58) 



[0446] In the event of calculating the mixture ratio a of the pixel included in the covered background region in the 
45 frame #n, the pixel values Bc1 through Bc9 of the pixels in the background region in the frame #n - 1 corresponding 
to the pixels in the frame #n, are used in Expression (50) through Expression (58). 

[0447] In Fig. 72, white circles indicate the pixels which are regarded as backgrounds, and solid circles indicate the 
pixels which are regarded as pixels in the mixed region. 

[0448] In the event of calculating the mixture ratio a of the pixel included in the uncovered background region shown 
so in Fig. 72, Expression (59) through Expression (67) described below may be formed. The pixel value of the pixel for 
calculation of the mixture ratio a is Mu5. 

Mu1 = (-1) • Bu1 • m + (-1) • Bu1 • q + Bu1 - p + (-1) • 
55 s + (-1)-t + u (59) 
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Mu2 = (0) ■ Bu2 • m + (-1) • Bu2 - q + Bu2 • p + (0) • s 

+ (-1)-t + u (60) 

Mu3 = (+1) • Bu3 • m + (-1) • Bu3 . q + Bu3 - p + (+1) • 

s + (-1).t + u (61) 

Mu4 = (-1) • Bu4 • m + (0) • Bu4 . q + Bu4 . p + (-1) • 

s + (0) • t + u (62) 

Mu5 = (0) . Bu5 • m + (0) • Bu5 • q + Bu5 . p + (0) . s 

+ (0) • t + u (63) 

Mu6 = (+1) • Bu6 • m + (0) • Bu6 • q + Bu6 • p + (+1) • 

S + (0) • t + u (64) 

Mu7 = (-1) • Bu7 • m + (+1) • Bu7 • q + Bu7 . p + (-1) • 

s + (+1)-t + u (65) 

Mu8 = (0) • Bu8 • m + (+1) ■ Bu8 ■ q + Bu8 • p + (0) ■ s 

+ (+1).t + u (66) 

Mu9 = (+1) • Bu9 • m + (+1) • Bu9 . q + Bu9 • p + (+1) • 

s + (+1) . t + u (67) 



[0449] In the event of calculating the mixture ratio a of the pixel included in the uncovered background region in the 
frame #n, the pixel values Bu1 through Bu9 of the pixels in the background region in the frame #n + 1 corresponding 
to the pixels in the frame #n, are used in Expression (59) through Expression (67). 

[0450] Fig. 73 is a block diagram which illustrates the configuration of the estimated mixture ratio processing unit 
45 401 . The image input to the estimated mixture ratio processing unit 401 is supplied to a delay circuit 501 and an addition 
unit 502. 

[0451] The delay circuit 501 delays the input image by one frame, and supplies to the addition unit 502. At the point 
that the frame #n is input to the addition unit 502 as an input image, the delay circuit 501 supplies the frame #n - 1 to 
the addition unit 502. 

so [0452] The addition unit 502 sets the pixel values of the pixels near the pixel for calculation of the mixture ratio a, 
and the pixel values of the frame #n - 1, for the normal equation. For example, the addition unit 502 sets the pixel 
values Mc1 through Mc9, and the pixel values Bc1 through Bc9 for the normal equation based upon Expression (50) 
through Expression (58). The addition unit 502 supplies the normal equation for which the pixel values have been set, 
to a computation unit 503. 

55 [0453] The computation unit 503 obtains the estimated mixture ratio by solving the normal equation supplied from 
the addition unit 502 by the sweeping method or the like, and outputs the obtained estimated mixture ratio. 
[0454] As described above, the estimated mixture ratio processing unit 401 can calculate the estimated mixture ratio 
based upon the input image, and supply to the mixture ratio decision unit 403. 
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[0455] Note that the estimated mixture ratio processing unit 402 has the same configuration as the estimated mixture 
ratio processing unit 401 , and accordingly description thereof will be omitted. 

[0456] Fig. 74 is a diagram which illustrates an example of the estimated mixture ratio calculated by the estimated 
mixture ratio processing unit 401 . Fig. 74 illustrates the estimated mixture ratio with regard to one line, wherein the 
5 movement v of the foreground corresponding to the object which moves at a constant velocity is 11 , and the results 
are calculated by the expression generated in increments of blocks 7x7 pixels. 

[0457] It can be understood that the estimated mixture ratio changes generally linearly in the mixed region as shown 
in Fig. 60. 

[0458] The mixture ratio decision unit 403 sets the mixture ratio based upon the region information indicating which 
10 of the foreground region, the background region, the covered background region, or the uncovered background region 
the pixel for calculation of the mixture ratio belongs to, supplied from the region specifying unit 1 01 . In the event that 
the pixel which is the object belongs to the foreground region, the mixture ratio decision unit 403 sets the mixture ratio 
to 0, in the event that the pixel which is the object belongs to the background region, sets the mixture ratio to 1, in the 
event that the pixel which is the object belongs to the covered background region, sets the mixture ratio to the estimated 
15 mixture ratio supplied from the estimated mixture ratio processing unit 401 , and in the event that the pixel which is the 
object belongs to the uncovered background region, sets the mixture ratio to the estimated mixture ratio supplied from 
the estimated mixture ratio processing unit 402. The mixture ratio decision unit 403 outputs the mixture ratio which is 
set based upon the region information. 

[0459] Referring to the flowchart shown in Fig. 75, the processing for calculation of the mixture ratio by the mixture 
20 ratio calculating unit 102 in a case that the estimated mixture ratio processing unit 401 has a configuration shown in 
Fig. 73 will be described. In Step S501 , the mixture ratio calculating unit 102 obtains the region information supplied 
from the region specifying unit 1 01 . In Step S502, the estimated mixture ratio processing unit 401 performs the process : 
ing of mixture ratio estimation, by a model corresponding to the covered background region, and supplies the estimated 
mixture ratio to the mixture ratio decision unit 403. Details of the processing for mixture ratio estimation will be described 
25 later with reference to the flowchart shown in Fig. 76. 

[0460] In Step S503, the estimated mixture ratio processing unit 402 performs the processing of mixture ratio esti- 
mation by a model corresponding to the uncovered background region, and supplies the estimated mixture ratio to the 
mixture ratio decision unit 403. 

[0461] In Step S504, the mixture ratio calculating unit 1 02 judges whether or not the mixture ratio has been estimated 
30 for the entire frame, and in the event that judgment is made that the mixture ratio has not been estimated for the entire 
frame, the flow returns to Step S502, and the processing of mixture ratio estimation for the following pixel is performed. 
[0462] In Step S504, in the event that judgment is made that the mixture ratio has been estimated for the entire 
frame, the flow proceeds to Step S505, and the mixture ratio decision unit 403 sets the mixture ratio based upon the 
region information which indicates which of the foreground region, the background region, the covered background 
35 region, or the uncovered background region the pixel of calculation of the mixture ratio belongs to, supplied from the 
region specifying unit 1 01 . In the event that the pixel which is the object belongs to the foreground region, the mixture 
ratio decision unit 403 sets the mixture ratio to 0, in the event that the pixel which is the object belongs to the background 
region, sets the mixture ratio to 1 , in the event that the pixel which is the object belongs to the covered background 
region! sets the mixture ratio to the estimated mixture ratio supplied from the estimated mixture ratio processing unit 
40 401 , and in the event that the pixel which is the object belongs to the uncovered background region, sets the mixture 
ratio to the estimated mixture ratio supplied from the estimated mixture ratio processing unit 402, and processing ends. 
[0463] As described above, the mixture ratio calculating unit 1 02 can calculate the mixture ratio a which is the amount 
of features corresponding to each pixel based upon the region information supplied from the region specifying unit 1 01 
and the input image. 

45 [0464] Using the mixture ratio a enables the separation of the foreground components and the background compo- 
nents contained in the pixel value while leaving the information of movement blurring contained in the image corre- 
sponding to the moving object. 

[0465] Also, synthesizing an image based upon the mixture ratio a enables creation of an image containing accurate 
movement blurring corresponding to the speed of the object which moves as if image taking of the real world had been 
so performed again. 

[0466] The processing for the mixture ratio estimation by a model corresponding to the covered background region, 
which corresponds to Step S502 shown in Fig. 75, will now be described with reference to the flowchart shown in Fig. 76. 
[0467] in Step S521 , the addition unit 502 sets the pixel values contained in the input image and the pixels contained 
the image supplied from the delay circuit 501 for the normal equation corresponding to a model of the covered back- 
55 ground region. 

[0468] In Step S522, the estimated mixture ratio processing unit 401 judges whether or not setting of the pixel which 
is the object has ended, and in the event that judgment is made that the setting for the pixel which is the object has 
not ended, the flow returns to Step S521 , and the processing of setting of the pixel value for the normal equation is 
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repeated. 

[0469] In Step S522, in the event that judgment is made that setting of pixel values with regard to the pixel which is 
the object has ended, the flow proceeds to Step S523, and the computation unit 503 computes the estimated mixture 
ratio based upon the normal equation which the pixel values have been set for, and outputs the obtained estimated 
s mixture ratio. 

[0470] As described above, the estimated mixture ratio processing unit 401 can computes the estimated mixture 
ratio based upon the input image. 

[0471] The processing for mixture ratio estimation by a model corresponding to the uncovered background region 
in Step S153 shown in Fig. 75 is the same as the processing shown in the flowchart in Fig. 76 using the normal equation 

10 corresponding to a model of the uncovered background region, and accordingly, description thereof will be omitted. 
[0472] Note that while description has been made with an assumption that the object corresponding to the back- 
ground keeps still, the processing of obtaining the mixture ratio described above may be applied even if the image 
corresponding to the background contains movement. For example, in the event that the image corresponding to the 
background region moves uniformly, the estimated mixture ratio processing unit 401 shifts the entire image correspond- 

15 ing to the movement, and performs processing in the same manner as a case wherein the object corresponding to the 
background keeps still. Also, in the event that the image corresponding to the background contains different movement 
at each local position, the estimated mixture ratio processing unit 401 selects pixels corresponding to the movement 
as pixels corresponding to pixels belonging to the mixed region, and performs the above-described processing. 
[0473] The foreground/background separation unit 105 will now be described. Fig. 77 is a block diagram which ii- 

20 lustrates an example of the configuration of the foreground/background separation unit 105. The input image supplied 
to the foreground/background separation unit 105 is supplied to a separation unit 601 , a switch 602, and a switch 603. 
The region information indicating the covered background region and the uncovered background region, which is sup- 
plied from the region specifying unit 103, is supplied to the separation unit 601 . The region information indicating the 
foreground region is supplied to the switch 602. The region information indicating the background region is supplied 

25 to the switch 603. 

[0474] The mixture ratio a supplied from the mixture ratio calculation unit 1 04 is supplied to the separation unit 601 . 
[0475] The separation unit 601 separates the background components from the covered background region in the 
input image, as well as foreground components, based upon the region information which indicates the covered back- 
ground region and the mixture ratio a, and outputs the foreground component image in the covered background region 
30 which consists of the separated foreground components, and the background component image in the covered back- 
ground region which consists of the separated background components. 

[0476] The separation unit 601 separates the background components from the uncovered background region in 
the input image, as well as foreground components, based upon the region information which indicates the uncovered 
background region and the mixture ratio a, and outputs the foreground component image in the uncovered background 
35 region which consists of the separated foreground components, and the background component image in the uncov- 
ered background region which consists of the separated background components. 

[0477] In the event of inputting the pixel corresponding to the foreground region, the switch 602 is closed based upon 
the region information which indicates the foreground region, and outputs the image in the foreground region. 
[0478] In the event of inputting the pixel corresponding to the background region, the switch 603 is closed based 
40 upon the region information which indicates the background region, and outputs the image in the background region. 
[0479] Fig. 78A and Fig. 78B are diagrams which illustrate the input image input to the foreground/background sep- 
aration unit 105, and the foreground component image and the background component image, output from the fore- 
ground/background separation unit 1 05. 

[0480] Fig. 78A is a schematic diagram which illustrates the displayed image, and Fig. 78B is a model diagram 
45 wherein one line of pixels including pixels belonging to the foreground region, pixels belonging to the background 
region, and pixels belonging to the mixed region, corresponding to Fig. 7BA, develop over the time direction. 
[0481] As shown in Fig. 78A and 78B, the image in the background region output from the foreground/background 
separation unit 105 is made up of pixels belonging to the background region. 

[0482] As shown in Fig. 78A and 78B, the image in the foreground region output from the foreground/background 

50 separation unit 1 05 is made up of pixels belonging to the foreground region. 

[0483] The pixel value of the pixel in the uncovered background region is separated into the background components 
and the foreground components by the foreground/background separation unit 1 05. The separated background com- 
ponents make up the background component image in the uncovered background region, and the separated foreground 
components make up the foreground component image in the uncovered background region. 

55 [0484] The pixel value of the pixel in the covered background region is separated into the background components 
and the foreground components by the foreground/background separation unit 105. The separated background com- 
ponents make up the background component image in the covered background region, and the separated foreground 
components make up the foreground component image in the covered background region. 
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[0485] A description will now be made regarding the separation processing for the foreground components and the 
background components from the pixel belonging to the mixed region performed by the separation unit 601 . 
[0486] Fig. 79 is a model of an image which indicates two frames of the foreground components and the background 
components, including the foreground corresponding to the object which moves from the left to the right In the drawing. 
5 In the model of the image shown in Fig. 79, the movement amount v of the foreground is 4, and the virtual dividing 
number is 4. 

[0487] In the frame #n, the left-most pixel and the fourteenth through eighteenth pixels from the left are made up of 
only the background components, and belong to the background region. In the frame #n, the second through fourth 
pixels from the left are made up of the background components and the foreground components, and belong to the 
w uncovered background region. In the frame #n, the eleventh through thirteenth pixels from the left are made up of the 
background components and the foreground components, and belong to the covered background region. In the frame 
#n, the fifth through tenth pixels from the left are made up of only the foreground components, and belong to the 
foreground region. 

[0488] In the frame #n + 1 , the first through fifth pixels from the left and the eighteenth pixel from the left are made 
'5 up of only the background components, and belong to the background region. In the frame #n + 1 , the sixth through 
eighth pixels from the left contain the background components and the foreground components, and belong to the 
uncovered background region. In the frame #n + 1 , the fifteenth through seventeenth pixels from the left contain the 
background components and the foreground components, and belong to the covered background region. In the frame 
#n + 1 , the ninth through fourteen pixels from the left are made up of only the foreground components, and belong to 
20 the foreground region. 

[0489] Fig. 80 is a diagram which describes the processing for separation of the foreground components from the 
pixel belonging to the covered background region. In Fig. 80, a1 through a1 8 are the mixture ratios corresponding to 
the pixels in the frame #n, respectively. In Fig. 80, the fifteenth through seventeenth pixels from the left belong to the 
covered background region: 

25 [0490] The pixel value C1 5 of the fifteenth pixel from the left in the frame #n is represented in Expression (68). 



CIS = B15/v + F09/V + F08/v + FOl/v 
30 =.<xl5 • B15 + F09/v + F08/V +' F07/v 

= Ctl5 • P15 + F09/v + F08/V + F07/v (68) 

35 [0491] Here, ct15 denotes the mixture ratio of the fifteenth pixel from the left in the frame #n. P15 denotes the pixel 
value of the fifteenth pixel from the left in the frame #n - 1 . 

[0492] The sum f15 of the foreground components of the fifteenth pixel from the left in the frame #n is represented 
in Expression (69) based upon Expression (68). 

f15= F09/v + F08/v+F07/v 

= C15-ct15. P15 (69) 



45 [0493] Similarly, the sum f16 of the foreground components of the sixteenth pixel from the left in the frame #n Is 
represented in Expression (70), and the sum f17 of the foreground components of the seventeenth pixel from the left 
in the frame #n is represented in Expression (71 ). 

_ n f16 = C16-ot16- P16 (70) 



f17 = C17-a17.P17 (71) 

55 [0494] As described above, the foreground component fc contained in the pixel value C of the pixel belonging to the 
covered background region is calculated by Expression (72). 
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fc = C-a- P (72) 
[0495] P denotes the pixel value of the corresponding pixel in the previous frame. 

[0496] Fig. 81 is a diagram which describes the processing for separating the foreground components from the pixel 
belonging to the uncovered background region. In Fig. 81 , a1 through a18 denote the mixture ratio corresponding to 
the pixels in the frame #n, respectively. In Fig. 81 , the second through fourth pixels from the left belong to the uncovered 
background region. 

[0497] The pixel value C02 of the second pixel from the left in the frame #n is represented in Expression (73). 

C02 = B02/v + B02/v + B02/v + FOl/v 
= Ct2 • B02 + FOl/v 

= ct2 * N02 + FOl/v (73) 



[0498] Here, ct2 denotes the mixture ratio of the second pixel from the left in the frame #n. N02 denotes the pixel 
20 value of the second pixel from the left in the frame #n + 1 . 

[0499] The foreground component sum of the second pixel from the left in the frame #n, f02, is represented in Ex- 
pression (74) based upon Expression (73). 



f02= F01/V 

= C02 - a2 • N02 (74) 



[0500] Similarly, the foreground component sum of the third pixel from the left in the frame #n, f03, is represented in 
30 Expression (75), and the foreground component sum of the fourth pixel from the left in the frame #n, f04, is represented 
in Expression (76). 



f03 = C03 - a3 ■ N03 (75) 
f04 = C04 - a4 • N04 (76) 



[0501 ] As described above, the foreground component f u contained in the pixel value C of the pixel belonging to the 
40 uncovered background region is calculated by Expression (77). 

f u = C - a • N (77) 

45 [0502] N denotes the pixel value of the corresponding pixel in the following frame. 

[0503] As described above, the separation unit 601 can separate the foreground components and the background 
components from the pixel belonging to the mixed region based upon the information indicating the covered background 
region and the information indicating the uncovered background region, which is included in the region information, 
and the mixture ratio a for each pixel. 

so [0504] Fig. 82 is a block diagram which illustrates an example of the configuration of the separation unit 601 for 
performing the processing described above. The image input to the separation unit 601 is supplied to frame memory 
621 , and the region information indicating the covered background region and the uncovered background region sup- 
plied from the mixture ratio calculating unit 104, and the mixture ratio a is input to a separation processing block 622. 
[0505] The frame memory 621 stores the input image in increments of frames. In the event that the object of process- 

55 ing is the frame #n, the frame memory 621 stores the frame #n - 1 which is the frame previous to the frame #n, frame 
#n t and the frame #n + 1 which is the frame following the frame #n. 

[0506] The frame memory 621 supplies the corresponding pixels in the frame #n - 1 , the frame #n, and the frame 
#n + 1 to the separation processing block 622. 
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[0507] The separation processing block 622 separates the foreground components and the background components 
from the pixel belonging to the mixed region in the frame #n by applying the computation described with reference to 
Fig. 80 and Fig. 81 to the pixel values of the pixels corresponding to the frame #n - 1 , the frame #n, and the frame #n 
+ 1 supplied from the frame memory 621 based upon the region information which indicates the covered background 
s region and the uncovered background region and the mixture ratio a. 

[0508] The separation processing block 622 comprises an uncovered region processing unit 631 and a covered 
region processing unit 632. 

[0509] A multiplication device 641 of the uncovered region processing unit 631 multiplies the pixel value of the pixel 
of the frame #n + 1 supplied from the frame memory 621 by the mixture ratio a, and outputs to a switch 642. In the 
io event that the pixel (corresponding to the pixel in the frame #n + 1) in the frame #n supplied from the frame memory 
621 belongs to the uncovered background region, the switch 642 is closed, and supplies the pixel value multiplied by 
the mixture ratio a supplied from the multiplication device 641 to a computation device 643. The value wherein the 
pixel value of the pixel in the frame #n + 1 is multiplied by the mixture ratio a supplied from the switch 642, is the same 
as the background component of the pixel value of the pixel corresponding to the frame #n, and is output as a back- 
's ground component image in the uncovered background region. 

[051 0] The computation device 643 obtains the foreground components by subtracting the background components 
supplied from the switch 642 from the pixel value of the pixel of the frame #n supplied from the frame memory 621 . 
The computation device 643 outputs the foreground component image made up of pixels in the frame #n belonging to 
the uncovered background region, 
20 [051 1 ] A multiplication device 651 of the covered region processing unit 632 multiplies the pixel value of the pixel of 
the frame #n - 1 supplied from the frame memory 621 by the mixture ratio a, and outputs to a switch 652. In the event 
that the pixel (corresponding to the pixel in the frame #n - 1) in the frame #n supplied from the frame memory 621 
belongs to the covered background region, the switch 652 is closed, and supplies the pixel value multiplied by the 
mixture ratio a supplied from the multiplication device 651 to a computation device 653. The value wherein the pixel 
25 value of the pixel in the frame #n - 1 is multiplied by the mixture ratio a supplied from the switch 652, is the same as 
the background component of the pixel value of the corresponding pixel in the frame #n, and is output as a background 
component image in the covered background region. 

[0512] The computation device 653 obtains the foreground components by subtracting the background components 
supplied from the switch 652 from the pixel value of the pixel of the frame #n supplied from the frame memory 621 . 
30 The computation device 653 outputs the foreground component image made up of pixels in the frame #n belonging to 
the covered background region. 

[0513] Using the mixture ratio a which is the amount of features enables entire separation of the foreground com- 
ponent and background component, contained in the pixel value. 

[0514] The separation processing for the foreground and the background by the foreground/background separation 
35 unit 1 05 will now be described, with reference to the flowchart shown in Fig. 83. In Step S601 , the frame memory 621 
of the separation unit 601 obtains the input image, and stores the frame #n which is the object of the separation of the 
foreground and the background, as well as the previous frame #n - 1 and the following frame #n + 1 . 
[0515] In Step S602, the separation processing block 622 of the separation unit 601 obtains the region information 
supplied from the mixture ratio calculation unit 1 04. In Step S603, the separation processing block 622 of the separation 
40 unit 601 obtains the mixture ratio a supplied from the mixture ratio calculation unit 104. 

[0516] In Step S604, the uncovered region processing unit 631 extracts the background components from the pixel 
value of the pixel belonging to the uncovered background region supplied from the frame memory 621 based upon the 
region information and the mixture ratio a, and outputs as the background component image in the uncovered back- 
ground region. 

<5 [0517] In Step S605, the uncovered region processing unit 631 extracts the foreground components from the pixel 
value of the pixel belonging to the uncovered background region supplied from the frame memory 621 based upon the 
region information and the mixture ratio a, and outputs as the foreground component image in the uncovered back-, 
ground region. 

[0518] In Step S606, the covered region processing unit 632 extracts the background components from the pixel 
50 value of the pixel belonging to the covered background region supplied from the frame memory 621 based upon the 
region information and the mixture ratio a, and outputs as the background component image in the covered background 
region. 

[0519] In Step S607, the covered region processing unit 632 extracts the foreground components from the pixel 
value of the pixel belonging to the covered background region supplied from the frame memory 621 based upon the 
55 region information and the mixture ratio a, outputs as the foreground component image in the covered background 
region, and the processing ends. 

[0520] As described above, the foreground/background separation unit 1 05 can separate the foregrou nd components 
and the background components from the input image based upon the region information and the mixture ratio a, and 
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output the foreground component image which is made up of only the foreground components, and the background 
component image which is made up of onfy the background components. 

[0521] Fig. 84 is a block diagram which illustrates the configuration of the separated image processing unit 106 for 
generating a coefficient set which is used for the class classification adaptation processing for generating an even 
5 higher resolution image in the spatial direction. For example, the separated image processing unit 106 of which con- 
figuration is shown in Fig. 84 generates a coefficient set which is used for the class classification adaptation processing 
for generating an HD image from an SD image based upon the input HD image. 

[0522] Background region tutor image frame memory 701 stores the image in the background region in the tutor 
image supplied from the foreground/background separation unit 1 05. The background region tutor image frame memory 
10 701 supplies the stored image in the background region in the tutor image to a weighted averaging unit 707-1 and a 
learning unit 714-1. 

[0523] Uncovered background region background component tutor image frame memory 702 stores the background 
component image in the uncovered background region in the tutor image supplied from the foreground/background 
separation unit 1 05. The uncovered background region background component tutor image frame memory 702 supplies 
15 the stored background component image in the uncovered background region.in the tutor image to a weighted aver- 
aging unit 707-2 and a learning unit 714-2. 

[0524] Uncovered background region foreground component tutor image frame memory 703 stores the foreground 
component image in the uncovered background region in the tutor image supplied from the foreground/background 
separation unit 1 05. The uncovered background region foreground component tutor image frame memory 703 supplies 
20 the stored foreground component image in the uncovered background region in the tutor image to a weighted averaging 
unit 707-3 and a learning unit 714-4. 

[0525] Covered background region background component tutor image frame memory 704 stores the background 
component image in the covered background region in the tutor image supplied from the foreground/background sep- 
aration unit 1 05. The covered background region background component tutor image frame memory 704 supplies the 
25 stored background component image in the covered background region in the tutor image to a weighted averaging 
unit 707-4 and a learning unit 71 4-4. 

[0526] Covered background region foreground component tutor image frame memory 705 stores the foreground 
component image in the covered background region in the tutor image supplied from the foreground/background sep- 
aration unit 105. The covered background region foreground component tutor image frame memory 705 supplies the 
30 stored foreground component image in the covered background region in the tutor image to a weighted averaging unit 
707-5 and a learning unit 714-5. 

[0527] Foreground region tutor image frame memory 706 stores the image in the foreground region in the tutor image 
supplied from the foreground/background separation unit 105. The foreground region tutor image frame memory 706 
supplies the stored foreground image in the tutor image to a weighted averaging unit 707-6 and a learning unit 714-6. 
35 [0528] The weighted averaging unit 707-1 generates an SD image which is a student image by 1/4 weighted-aver- 
aging the image in the background region in the tutor image which is an HD image, for example, supplied from the 
background region tutor image frame memory 701 , and supplies the generated SD image to background region student 
image frame memory 708. 

[0529] For example, the weighted averaging unit 707-1 takes four pixels of 2 x 2 (width x height) (which are portions 
40 represented by white circles in the drawing) as one increment in the tutor image as shown in Fig. 85, adds the pixel 
values of four pixels in each increment, and the sum is divided by 4. The weighted averaging unit 707-1 sets the 1/4 
weighted averaged results described above for the pixel of the student image positioned at the center of each increment 
(which are the portions represented by solid circles in the drawing). 

[0530] The background region student image frame memory 708 stores the student image corresponding to the 
45 image in the background region in the tutor image supplied from the weighted averaging unit 707-1 . The background 
region student image frame memory 708 supplies the stored student image corresponding to the image in the back- 
ground region in the tutor image to the learning unit 71 4-1 . 

[0531] The weighted averaging unit 707-2 generates an SD image which is a student image by 1/4 weighted-aver- 
aging the background component image in the uncovered background region in the tutor image which is an HD image, 
50 for example, supplied from the uncovered background region background component tutor image frame memory 702, 
and supplies the generated SD image to uncovered background region background component student image frame 
memory 709. 

[0532] The uncovered background region background component student image frame memory 709 stores the stu- 
dent image, which is an SD image, corresponding to the background component image in the uncovered background 
55 region in the tutor image supplied from the weighted averaging unit 707-2. The uncovered background region back- 
ground component student image frame memory 709 supplies the stored student image corresponding to the back- 
ground component image in the uncovered background region in the tutor image to the learning unit 714-2. 
[0533] The weighted averaging unit 707-3 generates an SD image which is a student image by 1/4 weighted-aver- 
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aging the foreground component image in the uncovered background region in the tutor image which is an HD image, 
for example, supplied from the uncovered background region foreground component tutor image frame memory 703, 
and supplies the generated SD image to uncovered background region foreground component student image frame 
memory 710. 

5 [0534] The uncovered background region foreground component student image frame memory 710 stores the stu- 
dent image, which is an SD image, corresponding to the foreground component image in the uncovered background 
region in the tutor image supplied from the weighted averaging unit 707-3. The uncovered background region fore- 
ground component student image frame memory 710 supplies the stored student image corresponding to the fore- 
ground component image in the uncovered background region in the tutor image to the learning unit 714-3. 

10 [0535] The weighted averaging unit 707-4 generates an SD image, which is a student image, by 1/4 weighted-av- 
eraging the background component image in the covered background region in the tutor image, for example, supplied 
from the covered background region background component tutor image frame memory 704, and supplies the gener- 
ated SD image to covered background region background component student image frame memory 711 . 
[0536] The covered background region background component student image frame memory 71 1 stores the student 

15 image, which is an SD image, corresponding to the background component image in the covered background region 
in the tutor image supplied from the weighted averaging unit 707-4. The covered background region background com- 
ponent student image frame memory 71 1 supplies the stored student image corresponding to the background compo- 
nent image in the covered background region in the tutor image to the learning unit 714-4. 

[0537] The weighted averaging unit 707-5 generates an SD image, which is a student image, by 1/4 weighted-av- 
20 eraging the foreground component image in the covered background region in the tutor image, for example, supplied 
from the covered background region foreground component tutor image frame memory 705, and supplies the generated 
SD image to covered background region foreground component student image frame memory 712. 
[0538] The covered background region foreground component student image frame memory 712 stores the student 
image, which is an SD image, corresponding to the foreground component image in the covered background region 
25 in the tutor image supplied from the weighted averaging unit 707-5. The covered background region foreground com- 
ponent student image frame memory 712 supplies the stored student image corresponding to the foreground compo- 
nent image in the covered background region in the tutor image to the learning unit 714-5. 

[0539] The weighted averaging unit 707-6 generates an SD image which is a student image by 1/4 weighted-aver- 
aging the image in the foreground region in the tutor image which is an HD image, for example, supplied from the 
30 foreground region tutor image frame memory 706, and supplies the generated SD image to foreground region student 
image frame memory 713. 

[0540] The foreground region student image frame memory 713 stores the student image, which is an SD image, 
corresponding to the image in the foreground region in the tutor image supplied from the weighted averaging unit 707-6. 
The foreground region student image frame memory 71 3 supplies the stored student image corresponding to the image 
35 jn the foreground region in the tutor image to the learning unit 714-6. 

[0541] The learning unit 714-1 generates a coefficient set corresponding to the background region based upon the 
image in the background region in the tutor image supplied from the background region tutor image frame memory 

701 and the student image corresponding to the image in the background region in the tutor image supplied from the 
background region student image frame memory 708, and supplies the generated coefficient set to coefficient set 

40 memory 715. 

[0542] The learning unit 71 4-2 generates a coefficient set corresponding to the background component image in the 
uncovered background region based upon the background component image in the uncovered background region in 
the tutor image supplied from the uncovered background region background component tutor image frame memory 

702 and the student image corresponding to the background component image in the uncovered background region 
45 in the tutor image supplied from the uncovered background region background component student image frame mem- 
ory 709, and supplies the generated coefficient set to the coefficient set memory 71 5. 

[0543] The learning unit 71 4-3 generates a coefficient set corresponding to the foreground component image in the 
uncovered background region based upon the foreground component image in the uncovered background region in 
the tutor image supplied from the uncovered background region foreground component tutor image frame memory 
so 703 and the student image corresponding to the foreground component image in the uncovered background region in 
the tutor image supplied from the uncovered background region foreground component student image frame memory 
909, and supplies the generated coefficient set to the coefficient set memory 715. 

[0544] The learning unit 71 4-4 generates a coefficient set corresponding to the background component image in the 
covered background region based upon the background component image in the covered background region in the 
55 tutor image supplied from the covered background region background component tutor image frame memory 704 and 
the student image corresponding to the background component image in the covered background region in the tutor 
image supplied from the covered background region background component student image frame memory 711 , and 
supplies the generated coefficient set to the coefficient set memory 71 5. 
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[0545] The learning unit 71 4-5 generates a coefficient set corresponding to the foreground component image in the 
covered background region based upon the foreground component image in the covered background region in the 
tutor image supplied from the covered background region foreground component tutor image frame memory 705 and 
the student image corresponding to the foreground component image in the covered background region in the tutor 
5 image supplied from the covered background region foreground component student image frame memory 712, and 
supplies the generated coefficient set to the coefficient set memory 715. 

[0546] The learning unit 714-6 generates a coefficient set corresponding to the foreground region based upon the 
image in the foreground region in the tutor image supplied from the foreground region tutor image frame memory 706 
and the student image corresponding to the image in the foreground region in the tutor image supplied from the fore- 
10 ground region student image frame memory 713, and supplies the generated coefficient set to the coefficient set mem- 
ory 715. 

[0547] The coefficient set memory 715 stores the coefficient set corresponding to the background region supplied 
from the learning unit 714-1 , the coefficient set corresponding to the background component image in the uncovered 
background region supplied from the learning unit 71 4-2, the coefficient set corresponding to the foreground component 

'5 image in the uncovered background region supplied from the learning unit 714-3, the coefficient set corresponding to 
the background component image in the covered background region supplied from the learning unit 714-4, the coef- 
ficient set corresponding to the foreground component image in the covered background region supplied from the 
learning unit 71 4-5, and the coefficient set corresponding to the foreground region supplied from the learning unit 71 4-6. 
[0548] In the event that there is no need to differentiate between the learning unit 714-1 through the learning unit 

20 714-6, individually, these will be simply referred to as a learning unit 714 below. 

[0549] Fig. 86 is a block diagram which illustrates the configuration of the learning unit 714. 
[0550] A class classification unit 731 comprises a class tap obtaining unit 751 and a waveform classification unit 
752, and classifies the pixel of interest of the input student image. The class tap obtaining unit 751 obtains a prede- 
termined number of class taps which are pixels of the student image corresponding to the pixel of interest, and supplies 

25 the obtained class taps to the waveform classification unit 752. 

[0551] For example, in Fig. 85, in the event that the pixel which is the ith from the top and the j'th from the left in the 
student image (which is a portion indicated by a solid circle in the drawing) is represented by X,j, the class tap obtaining 
unit 751 obtains a class tap which consists of nine pixels in total, i.e., the eight pixels at left-top, right-top, left, right, 
bottom-left, bottom, and right-bottom, adjacent to the pixel of interest Xg, X (M)(J . 1)( X (M) j, X (M)(j+1)l X^, X j(J+1) , X^^, 

30 X (l . 1)jp and X (l+1)0+1) , and also the pixel of interest. The class tap is supplied to the waveform classification unit 752. 
[0552] Note that in this case, while the class tap consists of a square-shaped block made up of 3 x 3 pixels, this 
needs not be a square; rather other arbitrary shapes may be used, for example, a rectangle-shape, a cross-shape, or 
the like. Also, the number of pixels making up the class tap is not restricted to nine pixels of 3 x 3 pixels. 
[0553] The waveform classification unit 752 performs class classification processing wherein the input Signals are 

35 classified into several classes based upon the features thereof, and classifies the pixel of interest into one class based 
upon the class taps. For example, the waveform classification unit 752 classifies the pixel of interest into one of 512 
classes, and supplies the class No. corresponding to the classified class to a prediction tap obtaining unit 732. 
[0554] Here, the class classification processing will now be described briefly. 

[0555] Now, let us say that a given pixel of interest and three adjacent pixels make up a class tap which consists of 
40 2x2 pixels as shown in Fig. 87A, and each pixel is represented by 1 bit (has a level of either 0 or 1 ). In Fig. 87A ( the 
solid circle denotes the pixel of interest. In this case, four pixel block of 2 x 2 pixels containing the pixel of interest can 
be classified into 1 6 (=(2 1 ) 4 ) patterns by the level distribution for each pixel as shown in Fig. 87B. In Fig. 87B, white 
circles denote 0, and solid circles denote 1 . Accordingly, in this case, the pixel of interest can be classified into sixteen 
patterns, wherein pattern -classification is the class-classification processing, and the processing is performed by the 
45 class classification unit 731 . 

[0556] Here, each pixel is generally appropriated around 8 bits. Also, with the present embodiment, the class tap 
consists of nine pixels of 3 x 3 pixels as described above. Accordingly, performing class classification processing for 
such a class tap as an object, the class tap would result the class tap being classified into a great number of classes 
of which number is (2 8 ) 9 . 

50 [0557] Accordingly, with the present embodiment, the ADCR processing is performed for the class tap by the wave- 
form classification unit 752, and this reduces the number of classes by reducing the number of bits of the pixels making 
up the class tap. 

[0558] In order to simplify description, the maximum value of the pixel value MAX and the minimum value of the pixel 
value MIN are detected in the ADRC processing with a class tap which consists of four pixels arrayed in a line as shown 
55 jn Fig. B8A. DR = MAX - MIN is then taken as the local dynamic range in the block which consists of a class tap, and 
the pixel values of the pixels making up the block of the class tap is re-quanttzed into K bits based upon the dynamic 
range DR. 

[0559] That is to say, the minimum value MIN is subtracted from each pixel value within the block, and the subtraction 
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E[y] = w 1 x t +w 2 x 2 + .... (78) 

[0574J Accordingly, for generalization, upon defining the matrix W which consists of a set of the prediction coefficients 
w, the matrix X which consists of a set of the student data, and the matrix V which consists of a set of the predicted 
values E[y) as 
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the following observation expression holds. 

XW = Y* (79) 

[0575] Let us now consider obtaining the predicted value E[y] near the pixel value y of the original image by applying 
the least square method to the observation expression. In this case, upon defining the matrix Y which consists of a set 
of pixel values y in the original image (which will be referred to as tutor data as appropriate) and the matrix E which 
consists of a set of the residuals e of the predicted values E[y] with regard to the pixel values y in the original image as 
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the following residual expression holds from Expression (79). 

XW=Y+E (80) 

[0576] In this case, the prediction coefficients w, for obtaining the predicted value E[y] near the pixel value y in the 
original image can be obtained by minimizing the squared margin of error 

m 

i=7 
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[0577] Accordingly, in a case that the derivative of the above-described squared margin of error from the prediction 
coefficient w, is 0, that is to say, in a case that the prediction coefficient w, satisfies the following expression, the pre- 
diction coefficient w, is the optimal value for obtaining the predicted values E[y] near the pixel value y in the original 
image. 

de 1 9e P de_ 

e ^r^4 + ^ e ^r 0(h1 ' 2 '" ,n) (81) 

[0578] Here, the following expression holds by differentiating Expression (80) by the prediction coefficient w h 

w? x "> ^2 = **• - = x,n - ° B 1t 2 - m) (82) 

[0579] Expression (83) is obtained from Expression (81) and Expression (82). 
m m m 

2,eiXu=0, ^eixi2=0,<~^eixm = 0 (83) 

i=7 i=J /=7 

[0580] Moreover, taking the relationship between the student data x, the prediction coefficient w, the tutor data y, 
and the residuals e in the residual expression (80), into consideration, the following normal equations can be obtained 
from Expression (83). 



(£jXilXa)Wi + (Z*XuXt2)w 2 + - • H-(2J XixXin)W n = (2jX a yi) 
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(84) 

[0581] As many normal equations represented in Expression (84) can be formed as the number of the prediction 
coefficients W which are to be obtained, and accordingly the optimal prediction coefficients w can be obtained by 
solving Expression (84). Note that Expressions (84) can be solved by applying the sweeping method (Gauss-Jordan 
elimination), for example. 

[0582] The adaptation processing consists of the optimal prediction coefficients w being obtained for each class, and 
the predicted values E[y] near the pixel values y in the tutor image being obtained by Expression (78) using the prediction 
coefficients w. 

[0583] The normal equation generating unit 734 generates the normal equations for calculating the optimal prediction 
coefficients w for each class, and the coefficient calculation unit 735 calculates the prediction coefficients w based 
upon the generated normal equations. 

[0584] Note that the adaptation processing is different from the interpolation processing with regard to the compo- 
nents which are not contained in the thinned out image and are contained in the original image being reproduced. That 
is to say, while in the event of taking only Expression (78) into consideration, the adaptation processing is the same 
as the interpolation processing using the interpolation filter, the prediction coefficients w corresponding to the tap co- 
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efficients of the interpolation filter is obtained by learning as if it were, using the tutor data y, and accordingly the 
adaptation processing can reproduce the components contained in the original image. Accordingly, it can be said that 
the adaptation processing acts to create an image, as if it were. 

[0585] Fig. 89 is a diagram which describes a coefficient set generated by the separated image processing unit 106 
of which configuration is shown in Fig. 84. The region specifying unit 103 specifies the foreground region, the back- 
ground region, the covered background region, and the uncovered background region in the input image. 
[0586] The input image wherein the regions have been specified and the mixture ratio a has been detected by the 
mixture ratio calculation unit 104, is separated into the image in the foreground region, the image in the background 
region, the foreground component image in the covered background region, the background component image in the 
covered background region, the foreground component image in the uncovered background region, and the back- 
ground component image in the uncovered background region, by the foreground/background separation unit 105. 
[0587] The separated image processing unit 1 06 calculates a coefficient set corresponding to the image in the fore- 
ground region, a coefficient set corresponding to the image in the background region, a coefficient set corresponding 
to the foreground component image in the covered background region, a coefficient set corresponding to the back- 
ground component image in the covered background region, a coefficient set corresponding to the foreground com- 
ponent image in the uncovered background region, and a coefficient set corresponding to the background component 
image in the uncovered background region individually, based upon the separated images, i.e., the image in the fore- 
ground region, image in the background region, foreground component image in the covered background region, back- 
ground component image in the covered background region, foreground component image in the uncovered back- 
ground region, and background component image in the uncovered background region. 

[0588] That is to say, the learning unit 714-1 calculates a coefficient set corresponding to the background region 
based upon the separated image in the background region, the learning unit 714-2 calculates a coefficient set corre- 
sponding to the background component image in the uncovered background region based upon the separated back- 
ground component image in the uncovered background region, the learning unit 714-3 calculates a coefficient set 
corresponding to the foreground component image in the uncovered background region based upon the separated 
foreground component image in the uncovered background region, the learning unit 714-4 calculates a coefficient set 
corresponding to the background component image in the covered background region based upon the separated 
background component image in the covered background region, the learning unit 714-5 calculates a coefficient set 
corresponding to the foreground component image in the covered background region based upon the separated fore- 
ground component image in the covered background region, and the learning unit 714-6 calculates a coefficient set 
corresponding to the foreground region based upon the separated image in the foreground region. 
[0589] The coefficient set corresponding to the background region is used for prediction of the pixel value in the 
background region in the class classification adaptation processing for predicting pixel values. The coefficient set cor- 
responding to the background component image in the uncovered background region is used for prediction of the pixel 
value corresponding to the background component image in the uncovered background region in the class classification 
adaptation processing for predicting pixel values. The coefficient set corresponding to the foreground component image 
in the uncovered background region is used for prediction of the pixel value corresponding to the foreground component 
image in the uncovered background region in the class classification adaptation processing for predicting pixel values. 
[0590] The coefficient set corresponding to the background component image in the covered background region is 
used for prediction of the pixel value corresponding to the background component image in the covered background 
region in the class classification adaptation processing for predicting pixel values. The coefficient set corresponding 
to the foreground component image in the covered background region is used for prediction of the pixel value corre- 
sponding to the foreground component image in the covered background region in the class classification adaptation 
processing for predicting pixel values. 

[0591] The coefficient set corresponding to the foreground region is used for prediction of the pixel value in the 
foreground region in the class classification adaptation processing for predicting pixel values. 
[0592] The predicted image corresponding to the image in the background region, the predicted image corresponding 
to the background component image in the uncovered background region, the predicted image corresponding to the 
foreground component image in the uncovered background region, the predicted image corresponding to the back- 
ground component image in the covered background region, the predicted image corresponding to the foreground 
component image in the covered background region, and the predicted image corresponding to the image in the fore- 
ground region, are synthesized into a single predicted image. 

[0593] Referring to the flowchart shown in Fig. 90, description will be made with regard to the processing of learning 
for generating a coefficient set which is used for prediction of pixel values based upon the class classification adaptation 
processing by the separated image processing unit 106 of which configuration is shown in Fig. 84. 
[0594] In Step S701 , the weighted averaging units 707-1 through 707-6 generate a student images of the image in 
the background region, the image in the foreground region, the background component image in the uncovered back- 
ground region, the foreground component image in the uncovered background region, the background component 
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image in the covered background region, and the foreground component image in the covered background region. 
That is to say, the weighted averaging unit 707-1 generates a student image corresponding to the image in the back- 
ground region in the tutor image by 1/4 weighted-averaging of the image in the background region in the tutor image 
stored in the background region tutor image frame memory 701 , for example. 
5 [0595] The weighted averaging unit 707-2 generates a student image corresponding to the background component 
image in the uncovered background region in the tutor image by 1/4 weighted-averaging of the background component 
image in the uncovered background region in the tutor image stored in the uncovered background region background 
component tutor image frame memory 702, for example. 

[0596] The weighted averaging unit 707-3 generates a student image corresponding to the foreground component 
10 image in the uncovered background region in the tutor image by 1/4 weighted-averaging of the foreground component 
image in the uncovered background region in the tutor image stored in the uncovered background region foreground 
component tutor image frame memory 703, for example. 

[0597] The weighted averaging unit 707-4 generates a student image corresponding to the background component 
image in the covered background region in the tutor image by 1/4 weighted-averaging of the background component 
15 image in the covered background region in the tutor image stored in the covered background region background com- 
ponent tutor image frame memory 704, for example. 

[0598] The weighted averaging unit 707-5 generates a student image corresponding to the foreground component 
image in the covered background region in the tutor image by 1/4 weighted-averaging of the foreground component 
image in the covered background region in the tutor image stored in the covered background region foreground com- 
20 ponent tutor image frame memory 705, for example. 

[0599] The weighted averaging unit 707-6 generates a student image corresponding to the image in the foreground 
region in the tutor image by 1/4 weighted-averaging of the image in the foreground region in the tutor image stored in 
the foreground region tutor image frame memory 706, for example. 

[0600] In Step S702, the learning unit 714-1 generates a coefficient set corresponding to the background region 
25 based upon the image in the background region in the tutor image stored in the background region tutor image frame 
memory 701 and the student image corresponding to the image in the background region in the tutor image stored in 
the background region student image frame memory 708. Details of the processing for generating of a coefficient set 
in Step S702 will be described later with reference to the flowchart shown in Fig. 91 . 

[0601] In Step S703, the learning unit 714-2 generates a coefficient set corresponding to the background component 
30 image in the uncovered background region based upon the background component image in the uncovered background 
region in the tutor image stored in the uncovered background region background component tutor image frame memory 

702 and the student image corresponding to the background component image in the uncovered background region 
in the tutor image stored in the uncovered background region background component student image frame memory 
709. 

35 [0602] In Step S704, the learning unit 714-3 generates a coefficient set corresponding to the foreground component 
image in the uncovered background region based upon the foreground component image in the uncovered background 
region in the tutor image stored in the uncovered background region foreground component tutor image frame memory 

703 and the student image corresponding to the foreground component image in the uncovered background region in 
the tutor image stored in the uncovered background region foreground component student image frame memory 710. 

<o [0603] In Step S705, the learning unit 71 4-4 generates a coefficient set corresponding to the background component 
image in the covered background region based upon the background component image in the covered background 
region in the tutor image stored in the covered background region background component tutor image frame memory 

704 and the student image corresponding to the background component image in the covered background region in 
the tutor image stored In the covered background region background component student image frame memory 711 . 

45 [0604] In Step S706, the learning unit 714-5 generates a coefficient set corresponding to the foreground component 
image in the covered background region based upon the foreground component image in the covered background 
region in the tutor image stored in the covered background region foreground component tutor image frame memory 

705 and the student image corresponding to the foreground component image in the covered background region in 
the tutor image stored in the covered background region foreground component student image frame memory 712. 

so [0605] In Step S707, the learning unit 71 4-6 generates a coefficient set corresponding to the foreground region based 
upon the image in the foreground region in the tutor image stored in the foreground region tutor image frame memory 

706 and the student image corresponding to the image in the foreground region in the tutor image stored in the fore- 
ground region student image frame memory 713. 

[0606] In Step S708, the learning units 714-1 through the learning unit 712-4 output a coefficient set corresponding 
55 to the background region, a coefficient set corresponding to the background component image in the uncovered back- 
ground region, a coefficient set corresponding to the foreground component image in the uncovered background region, 
a coefficient set corresponding to the background component image in the covered background region, a coefficient 
set corresponding to the foreground component image in the covered background region, or a coefficient set corre- 
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spending to foreground region, to the coefficient set memory 715, respectively. The coefficient set memory 715 stores 
the coefficient sets each of which corresponds to the background region, the foreground region, the background com- 
ponent image in the uncovered background region, the foreground component image in the uncovered background 
region, the background component image in the covered background region, and the foreground component image in 
the covered background region, and then the processing ends. 

[0607] As described above, the separated image processing unit 1 06 of which configuration is shown in Fig. 84 can 
generate a coefficient set corresponding to the image in the background region, a coefficient set corresponding to the 
background component image in the uncovered background region, a coefficient set corresponding to the foreground 
component image in the uncovered background region, a coefficient set corresponding to the background component 
image in the covered background region, a coefficient set corresponding to the foreground component image in the 
covered background region, and a coefficient set corresponding to the image in the foreground region. 
[0608] Note that it is needless to say that the processing in Step S702 through Step S707 may be performed serially 
or in parallel. 

[0609] Referring to the flowchart shown in Fig. 91 , the processing for generating of a coefficient set corresponding 
to the background region performed by the learning unit 71 4-1 , corresponding to the processing in Step S702, will now 
be described. 

[0610] In Step S721 , the learning unit 714-1 judges whether or not there are any unprocessed pixels in the student 
image corresponding to the background region, and in the event that judgment is made that there are unprocessed 
pixels in the student image corresponding to the background region, the flow proceeds to Step S722, and the pixel of 
interest is obtained from the student image corresponding to the background region in raster scan sequence. 
[0611] In Step S723, the class tap obtaining unit 751 of the class classification unit 731 obtains a class tap corre- 
sponding to the pixel of interest from the student image stored in the background region student image frame memory 
708. In Step S724, the waveform classification unit 752 of the class classification unit 731 applies the ADRC processing 
to the class tap, this reduces the number of bits of pixels making up the class tap, and the pixel of interest is classified. 
In Step S725, the prediction tap obtaining unit 732 obtains a prediction tap corresponding to the pixel of interest from 
the student image stored in the background region student image frame memory 708 based upon the classified class. 
[0612] In Step S726, the corresponding pixel obtaining unit 733 obtains pixels corresponding to the pixel value which 
is to be predicted from the background region image of the tutor image stored in the background region tutor image 
frame memory 701 based upon the classified class. 

[0613] In Step S727, the norma! equation generating unit 734 adds the pixel values of pixels corresponding to the 
prediction tap and the pixel value which is to be predicted to the matrix for each class based upon the classified class, 
the flow returns to Step S721 , and the separated image processing unit 1 06 repeats judgment whether or not unproc- 
essed pixels exist. The prediction tap and the matrix for each class to which the pixel values of pixels corresponding 
to the prediction tap and the pixel value which is to be predicted is added, corresponds to the normal equations for 
calculating a coefficient set for each class. 

[0614] In Step S721 , in the event that judgment is made that there are no unprocessed pixels in the student image, 
the flow proceeds to Step S728, and the normal equation generating unit 734 supplies the matrix for each class for 
which the pixel values of the pixels corresponding to the prediction tap and the pixel value which is to be predicted are 
set, to the coefficient calculation unit 735. The coefficient calculation unit 735 calculates a coefficient set for each class 
corresponding to the background region by solving the matrix for each class, wherein the pixel values of pixels corre- 
sponding to the prediction tap and the pixel value which is to be predicted are set. 

[0615] Note that the coefficient set is not restricted to predicting the pixel values by linear prediction, rather, an 
arrangement may be made wherein the coefficient calculation unit 735 calculates a coefficient set for predicting the 
pixel values by non-linear prediction. 

[0616] In Step S729, the coefficient calculation unit 735 outputs the coefficient set for each class, corresponding to 
the background region to the coefficient set memory 715, and the processing ends. 

[061 7] As described above, the learning unit 71 4-1 can generate a coefficient set corresponding to the background 
region. 

[0618] The processing for generating the coefficient set corresponding to the background component image in the 
uncovered background region by the learning unit 714-2 corresponding to Step S703 is the same as the processing 
described with reference to the flowchart shown in Fig. 91 except for using the background component image in the 
uncovered background region stored in the uncovered background region background component tutor image frame 
memory 702 and the student image corresponding to the background component image in the uncovered background 
region stored in the uncovered background region background component student image frame memory 709, and 
accordingly, description thereof will be omitted. 

[061 9] The processing for generating of the coefficient set corresponding to the foreground component image in the 
uncovered background region by the learning unit 714-3 corresponding to Step S704 is the same as the processing 
described with reference to the flowchart shown in Fig. 91 except for using the foreground component image in the 
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uncovered background region stored in the uncovered background region foreground component tutor image frame 
memory 703 and the student image corresponding to the foreground component image in the uncovered background 
region stored in the uncovered background region foreground component student image frame memory 710, and ac- 
cordingly, description thereof will be omitted. 

[0620] The processing for generating of the coefficient set corresponding to the background component image in 
the covered background region by the learning unit 714-4 corresponding to Step S705 is the same as the processing 
described with reference to the flowchart shown in Fig. 91 except for using the background component image in the 
covered background region stored in the covered background region background component tutor image frame mem- 
ory 704 and the student image corresponding to the background component image in the covered background region 
stored in the covered background region background component student image frames memory 711 , and accordingly, 
description thereof will be omitted. 

[0621] The processing for generating of the coefficient set corresponding to the foreground component image in the 
covered background region by the learning unit 714-5 corresponding to Step S706 is the same as the processing 
described with reference to the flowchart shown In Fig. 91 except for using the foreground component image in the 
covered background region stored in the covered background region foreground component tutor image frame memory 

705 and the student image corresponding to the foreground component image in the covered background region stored 
in the covered background region foreground component student image frame memory 712, and accordingly, descrip- 
tion thereof will be omitted. 

[0622] The processing for generating of the coefficient set corresponding to the foreground region by the learning 
unit 714-6 corresponding to Step S707 is the same as the processing described with reference to the flowchart shown 
in Fig. 91 except for using the image in the foreground region stored in the foreground region tutor image frame memory 

706 and the student image corresponding to the image in the foreground region stored in the foreground region student 
. image frame memory 713, and accordingly, description thereof will be omitted. 

[0623] As described above, the separated image processing unit 1 06 of which configuration is shown in Fig. 84 can 
generate a coefficient set corresponding to the background region, a coefficient set corresponding to the background 
component image in the uncovered background region, a coefficient set corresponding to the foreground component 
image in the uncovered background region, a coefficient set corresponding to the background component image in 
the covered background region, a coefficient set corresponding to the foreground component image in the covered 
background region, and a coefficient set corresponding to the foreground region, individually. 
[0624] Fig. 92 is a block diagram which illustrates the configuration of the separated image processing unit 106 for 
generating an even higher resolution image in the spatial direction by performing the class classification adaptation 
processing. For example, the separated image processing unit 106 of which configuration is shown in Fig. 92 generates 
an HD image by performing the class classification processing based upon the input image which is an SD image. 
[0625] Background region frame memory 801 stores the image in the background region made up of pixels belonging 
to the background region supplied from the foreground/background separation unit 1 05. The background region frame 
memory 801 supplies the stored image in the background region to a mapping unit 807-1. 

[0626] Uncovered background region background component image frame memory 802 stores the background com- 
ponent image in the uncovered background region supplied from the foreground/background separation unit 1 05. The 
uncovered background region background component image frame memory 802 supplies the stored background com- 
ponent image in the uncovered background region to a mapping unit 807-2. 

[0627] Uncovered background region foreground component image frame memory 803 stores the foreground com- 
ponent image in the uncovered background region supplied from the foreground/background separation unit 1 05. The 
uncovered background region foreground component image frame memory 803 supplies the stored foreground com- 
ponent image in the uncovered background region to a mapping unit 807-3. 

[0628] Covered background region background component image frame memory 804 stores the background com- 
ponent image in the covered background region supplied from the foreground/background separation unit 105. The 
covered background region background component image frame memory 804 supplies the stored background com- 
ponent image in the covered background region to a mapping unit 807-4. 

[0629] Covered background region foreground component image frame memory 805 stores the foreground compo- 
nent image in the covered background region supplied from the foreground/background separation unit 105. The cov- 
ered background region foreground component image frame memory 805 supplies the stored foreground component 
image in the covered background region to a mapping unit 807-5. 

[0630] Foreground region frame memory 806 stores the image in the foreground region made up of pixels belonging 
to the foreground region supplied from the foreground/background separation unit 1 05. The foreground region image 
frame memory 806 supplies the stored image in the foreground region to a mapping unit 807-6. 
[0631] The mapping unit 807-1 generates a predicted image corresponding to the image in the background region 
stored in the background region frame memory 801 by the class classification adaptation processing based upon the 
coefficient set corresponding to the background region stored in coefficient set memory 808. The mapping unit 807-1 
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supplies the generated predicted image to a synthesizing unit 809. 

[0632] The mapping unit 807-2 generates a predicted image corresponding to the background component image in 
the uncovered background region stored in the uncovered background region background component image frame 
memory 802 by the class classification adaptation processing based upon the coefficient set corresponding to the 
s background component image in the uncovered background region stored in the coefficient set memory 808. The 
mapping unit 807-2 supplies the generated predicted image to the synthesizing unit 809. 

[0633] The mapping unit 807-3 generates a predicted image corresponding to the foreground component image in 
the uncovered background region stored in the uncovered background region foreground component image frame 
memory 803 by the class classification adaptation processing based upon the coefficient set corresponding to the 
w foreground component image in the uncovered background region stored in the coefficient set memory 808. The map- 
ping unit 807-3 supplies the generated predicted image to the synthesizing unit 809. 

[0634] The mapping unit 807-4 generates a predicted image corresponding to the background component image in 
the covered background region stored in the covered background region background component image frame memory 

804 by the class classification adaptation processing based upon the coefficient set corresponding to the background 
is component image in the covered background region stored in the coefficient set memory 808. The mapping unit 807-4 

supplies the generated predicted image to the synthesizing unit 809. 

[0635] The mapping unit 807-5 generates a predicted image corresponding to the foreground component image in 
the covered background region stored in the covered background region foreground component image frame memory 

805 by the class classification adaptation processing based upon the coefficient set corresponding to the foreground 
20 component image in the covered background region stored in the coefficient set memory 808. The mapping unit 807-5 

supplies the generated predicted image to the synthesizing unit 809. 

[0636] The mapping unit 807-6 generates a predicted image corresponding to the image in the foreground region 
stored in the foreground region frame memory 806 by the class classification adaptation processing based upon the 
coefficient set corresponding to the foreground region stored in the coefficient set memory 808. The mapping unit 807-6 

25 supplies the generated predicted image to the synthesizing unit 809. 

[0637] The synthesizing unit 809 synthesizes the predicted image corresponding to the image in the background 
region supplied from the mapping unit 807-1 , the predicted image corresponding to the background component image 
in the uncovered background region supplied from the mapping unit 807-2, the predicted image corresponding to the 
foreground component image in the uncovered background region supplied from the mapping unit 807-3, the predicted 

30 image corresponding to the backgrou nd component image in the covered background region supplied from the mapping 
unit 807-4, the predicted image corresponding to the foreground component image in the covered background region 
supplied from the mapping unit 807-5, and the predicted image corresponding to the image in the foreground region 
supplied from the mapping unit 807-6, and supplies the synthesized predicted image to frame memory 810. 
[0638] The frame memory 81 0 stores the predicted image supplied from the synthesizing unit 809, and also outputs 

35 the stored image as an output image. 

[0639] In the event that there is no need to differentiate the mapping unit 807-1 through the mapping unit 807-6 

individually, these will be simply referred to as the mapping unit 807. 

[0640] Fig. 93 is a block diagram which illustrates the configuration of the mapping unit 807. 

[0641] The mapping processing unit 831 comprises a class classification unit 841 for performing the class classifi- 

40 cation processing, a prediction tap obtaining unit 842 for performing the adaptation processing, and a prediction com- 
putation unit 843. 

[0642] The class classification unit 841 comprises a class tap obtaining unit 851 and a waveform classification unit 
852, and classifies the pixel of interest in the separated input image which is one of the image in the background region, 
the background component image in the uncovered background region, the foreground component image in the un- 

45 covered background region, the background component image in the covered background region, the foreground 
component image in the covered background region, or the image in the foreground region. 
[0643] The class tap obtaining unit 851 obtains a predetermined number of class taps corresponding to the pixel of 
interest of the separated input image, and supplies the obtained class taps to the waveform classification unit 852. For 
example, the class tap obtaining unit 851 obtains nine class taps, and supplies the obtained class taps to the waveform 

50 classification unit 852. 

[0644] The waveform classification unit 852 reduces the number of bits of the pixels making up the class taps by 
applying the ADRC processing to the class taps, classifies the pixel of interest into one of the predetermined number 
of classes, and supplies the class No. corresponding to the classified class to the prediction tap obtaining unit 842. 
For example, the waveform classification unit 852 classifies the pixel of interest to one of 512 classes, and supplies 
55 the class No. corresponding to the classified class to the prediction tap obtaining unit 842. 

[0645] The prediction tap obtaining unit 842 obtains the predetermined number of prediction taps corresponding to 
the class from the separated input image which is one of the image in the background region, the background com- 
ponent image in the uncovered background region, the foreground component image in the uncovered background 
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region, the background component image in the covered background region, the foreground component image in the 
covered background region, or the image in the foreground region, based upon the class No., and supplies the obtained 
class taps and class No. to the prediction computation unit 843. 

[0646] The prediction computation unit 843 obtains a coefficient set corresponding to the class and corresponding 
to the image which is to be predicted, from the coefficient sets corresponding to the background region, the background 
component image in the uncovered background region, the foreground component image in the uncovered background 
region, the background component image in the covered background region, the foreground component image in the 
covered background region, and the foreground region, stored in the coefficient set memory 808, based upon the class 
No. The prediction computation unit 843 predicts a pixel value in the predicted image by linear prediction based upon 
the coefficient set and the prediction taps corresponding to the class, and corresponding to the image which is to be 
predicted. The prediction computation unit 43 supplies the predicted pixel value to the frame memory 832. 
[0647] Note that an arrangement may be made wherein the prediction computation unit 843 predicts the pixel value 
in the predicted image by non-linear prediction. 

[0648] The frame memory 832 stores the predicted pixel values supplied from the mapping processing unit 831 , and 
outputs the image made up of the predicted pixel values. 

[0649] Referring to the images shown in Fig. 94A through Fig. 99B, description will be made with regard to the results 
of the processing of the image processing device according to the present invention having the separated image 
processing unit 106 of which configuration is shown in Fig. 92. 

[0650] In the processing for generating results shown by way of examples, the sum of the number of classes in the 
class classif ication adaptation processing in the image processing device of the present invention is approximately the 
same as the number of classes in the conventional class classification adaptation processing. That is to say, the number 
of classes in the conventional class classification adaptation processing is 2048, and the number of the classes in the 
class classification adaptation processing in the image processing device of the present invention corresponding to 
the images in each region is arranged to be 512. 

[0651] Also, the number of the prediction taps in the conventional class classification adaptation processing and the 
number of the prediction taps in the class classification adaptation processing for each region in the image processing 
device of the present invention, are 9, i.e., the same. 

[0652] Referring to Fig. 94A through Fig. 96B, the results of the prediction in the covered background region will be 
described. 

[0653] Fig. 94A is a diagram which illustrates an example of the image in the mixed region of the tutor image. Fig. 
94B is a diagram which indicates the change in pixel value corresponding to the position in the spatial direction in the 
image in the mixed region of the tutor image. 

[0654] Fig. 95A is a diagram which illustrates an example of the image in the mixed region generated by the con- 
ventional class classification adaptation processing corresponding to the tutor image illustrated in Fig. 94A. Fig. 95B 
is a diagram which indicates the change in pixel value corresponding to the position in the spatial direction in the image 
in the mixed region, generated by the conventional class classification adaptation processing, corresponding to the 
tutor image illustrated in Fig. 94. 

[0655] Fig. 96A is a diagram which illustrates an example of the image in the mixed region, generated by the sepa- 
rated image processing unit 106 of which configuration is shown in Fig. 92, corresponding to the tutor image shown in 
Fig. 94A. Fig. 95B is a diagram which indicates the change in pixel value corresponding to the position in the spatial 
direction in the image in the mixed region, generated by the separated image processing unit 1 06 of which configuration 
is shown in Fig. 92, corresponding to the tutor image shown in Fig. 94A. 

[0656] The pixel values in the image in the mixed region, generated by the conventional class classification adaptation 
processing, change in a stepped manner, as compared with the tutor image, and also are visually confirmed to change 
in a stepped manner in the actual generated image. 

[0657] Conversely, the pixel values in the image in the mixed region, generated by the separated image processing 
unit 1 06 of which configuration is shown in Fig. 92, change more smoothly as compared with conventional arrangement, 
and indicates change even closer to the tutor image. Also, in the event of visually confirming the image generated by 
the separated image processing unit 106, the image is confirmed to be an even smoother image as compared with 
conventional arrangement. 

[0658] The image in the mixed region, generated by the separated image processing unit 106 of which configuration 
is shown in Fig. 92, changes more smoothly as compared with the image generated by the input image being divided 
into the foreground region, mixed region, or background region. 

[0659] Referring to Fig. 97A through Fig. 99B, description will be made with regard to the results of the prediction in 
the foreground region wherein the pixel values change generally linearly with regard to the pixel position. 
[0660] Fig. 97A is a diagram which illustrates an example of the image in the foreground region in the tutor image 
wherein the pixel values change generally linearly. Fig. 97B is a diagram which indicates change in pixel value corre- 
sponding to the position in the spatial direction in the image in the foreground region of the tutor image wherein the 
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pixel values change generally linearly. 

[0661] Fig. 98A is a diagram which illustrates an example of the image in the foreground region, corresponding to 
the image shown in Fig. 97A, generated by the conventional class classification adaptation processing. Fig. 98B is a 
diagram which indicates the change in pixel value corresponding to the position in the spatial direction, in the image 
in the foreground region, corresponding to the image shown in Fig. 97A, generated by the conventional class classifi- 
cation adaptation processing. 

[0662] Fig. 99 A is a diagram which illustrates an example of the image in the foreground region corresponding to 
the image shown in Fig. 97A, generated by the separated image processing unit 106 of which configuration is shown 
in Fig. 92. Fig. 99B is a diagram which indicates the change in pixel value, corresponding to the position in the spatial 
direction, in the image in the foreground region, corresponding to the image shown in Fig. 97A, generated by the 
separated image processing unit 1 06 of which configuration is shown in Fig. 92. 

[0663] The pixel values in the image in the foreground region generated by the conventional class classification 
adaptation processing change in a stepped manner as compared with the tutor image in the same manner as the mixed 
region, and the change in a stepped manner can be visually recognized in the actual image. 
[0664] Conversely, the pixel values in the image in the foreground region generated by the separated image process- 
ing unit 1 06 of which configuration is shown in Fig. 92, change more smoothly as compared with conventional arrange- 
ment, and are extremely close to the values in the tutor image, in visual confirmation of the image generated by the 
separated image processing unit 1 06, the difference between the image and the tutor image could not be observed. 
[0665] Referring to the flowchart shown in Fig. 100, description will now be made with regard to the processing for 
creation of an image by the separated image processing unit 106 of which configuration is shown in Fig. 92. 
[0666] In Step S801 , the mapping unit 807-1 predicts an image corresponding to the image in the background region 
stored in the background region frame memory 801 by the class classification adaptation processing based upon the 
coefficient set corresponding to the background region stored in the coefficient set memory 808. 
[0667] Details of the processing for prediction of the image corresponding to the image in the background region will 
be described later with reference to the flowchart shown in Fig. 101 . 

[0668] In Step S802, the mapping unit 807-2 predicts an image corresponding to the background component image 
in the uncovered background region stored in the uncovered background region background component image frame 
memory 802 by the class classification adaptation processing based upon the coefficient set corresponding to the 
background component image in the uncovered background region stored in the coefficient set memory 808. 
[0669] In Step S803, the mapping unit 807-3 predicts an image corresponding to the foreground component image 
in the uncovered background region stored in the uncovered background region foreground component image frame 
memory 803 by the class classification adaptation processing based upon the coefficient set corresponding to the 
foreground component image in the uncovered background region stored in the coefficient set memory 808. 
[0670] In Step S804, the mapping unit 807-4 predicts an image corresponding to the background component image 
in the covered background region stored in the covered background region background component image frame mem- 
ory 804 by the class classification adaptation processing based upon the coefficient set corresponding to the back- 
ground component image in the covered background region stored in the coefficient set memory 808. 
[0671] in Step S805, the mapping unit 807-5 predicts an image corresponding to the foreground component image 
in the covered background region stored in the covered background region foreground component image frame memory 
605 by the class classification adaptation processing based upon the coefficient set corresponding to the foreground 
component image in the covered background region stored in the coefficient set memory 808. 
[0672] In Step S806, the mapping unit 807-6 predicts an image corresponding to the image in the foreground region 
stored in the foreground region frame memory 806 by the class classification adaptation processing based upon the 
coefficient set corresponding to the foreground region stored in the coefficient set memory 808. 
[0673] In Step S807, the synthesizing unit 809 synthesizes the predicted image corresponding to the image in the 
background region, the predicted image corresponding to the background component image in the uncovered back- 
ground region, the predicted image corresponding to the foreground component image in the uncovered background 
region, the predicted image corresponding to the background component image in the covered background region, 
the predicted image corresponding to the foreground component image in the covered background region, and the 
predicted image corresponding to the foreground region. The synthesizing unit 809 supplies the synthesized image to 
the frame memory 810. The frame memory 810 stores the image supplied from the synthesizing unit 809. 
[0674] In Step S808, the frame memory 810 outputs the stored synthesized image, and the processing ends. 
[0675] As described above, the image processing device having the separated image processing unit 106 of which 
configuration is shown in Fig. 92 can generate a predicted image for each of separated images, i.e., the image in the 
background region, the background component image in the uncovered background region, the foreground component 
image in the uncovered background region, the background component image in the covered background region, the 
foreground component image in the covered background region, and the image in the foreground region. 
[0676] Note that it is needless to say that the processing in Step S801 through Step S806 may be performed in serial 
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manner, as well as in a parallel manner. 

[0677] Referring to the flowchart shown in Fig. 1 01 , the processing for prediction of the image corresponding to the 
background region by the mapping unit 807-1 corresponding to Step S801 will be described. 
[0678] In Step S821 , the mapping unit 807-1 judges whether or not there are any unprocessed pixels in the back- 
ground region, and in the event that judgment is made that there are unprocessed pixels in the background region 
image, the flow proceeds to Step S822, and the mapping processing unit 831 obtains the coefficient set corresponding 
to the background region stored in the coefficient set memory 808. In Step S823, the mapping processing unit 831 
obtains a pixel of interest from the image of the background region stored in the background region frame memory 801 
in raster scan sequence. 

[0679] In Step S824, the class tap obtaining unit 851 of the class classification unit 841 obtains the class tap corre- 
sponding to the pixel of interest from the image in the background region stored in the background region frame memory 
801 . In Step S825, the waveform classification unit 852 of the class classification unit 841 reduces the number of bits 
of pixels making up the class tap by applying the ADRC processing to the class tap, and performs class classification 
for the pixel of interest. In Step S826, the predication tap obtaining unit 842 obtains the prediction tap corresponding 
to the pixel of interest from the image in the background region stored in the background region frame memory 801 
based upon the classified class. 

[0680] In Step S827, the prediction computation unit 843 predicts pixel values of the predicted image by linear pre- 
diction based upon the coefficient set and the prediction tap, corresponding to the background region and the classified 
class. 

[0681] Note that the prediction computation unit 843 may predict the pixel values of the predicted image by non- 
linear prediction, as well as by linear prediction. 

[0682] In Step S828, the prediction computation unit 843 outputs the predicted pixel value to the frame memory 832. 
The frame memory 832 stores the pixel value supplied from the prediction computation unit 843. The procedure returns 
to Step S821 , and judgment whether or not any unprocessed pixels exist is repeated. 

[0683] In Step S821, in the event that judgment is made that there is no unprocessed pixel in the image in the 
background region, the flow proceeds to Step S829, the frame memory 832 outputs the stored predicted image cor- 
responding to the image in the background region, and processing ends. 

[0684] As described above, the mapping unit 807-1 can predict the image corresponding to the image in the back- 
ground region based upon the image in the background region of the separated input image. 
[0685] The processing for generating of the predicted image corresponding to the background component image in 
the uncovered background region by the mapping unit 807-2 corresponding to Step S802 is the same as the processing 
described with reference to the flowchart shown in Fig. 101 except for using the background component image in the 
uncovered background region stored in the uncovered background region background component image frame mem- 
ory 802 and the coefficient set corresponding to the background component image in the uncovered background region, 
and accordingly, description thereof will be omitted. 

[0686] The processing for generating of the predicted image corresponding to the foreground component image in 
the uncovered background region by the mapping unit 807-3 corresponding to Step S803 is the same as the processing 
described with reference to the flowchart shown in Fig. 101 except for using the foreground component image in the 
uncovered background region stored in the uncovered background region foreground component image frame memory 

803 and the coefficient set corresponding to the foreground component image in the uncovered background region, 
and accordingly, description thereof will be omitted. 

[0687] The processing for generating of the predicted image corresponding to the background component image in 
the covered background region by the mapping unit 807-4 corresponding to Step S804 is the same as the processing 
described with reference to the flowchart shown in Fig. 101 except for using the background component image in the 
covered background region stored in the covered background region background component image frame memory 

804 and the coefficient set corresponding to the background component image in the covered background region, and 
accordingly, description thereof will be omitted. 

[0688] The processing for generating of the predicted image corresponding to the foreground component image in 
the covered background region by the mapping unit 807-5 corresponding to Step S805 is the same as the processing 
described with reference to the flowchart shown in Fig. 101 except for using the foreground component image in the 
covered background region stored in the covered background region foreground component image frame memory 805 
and the coefficient set corresponding to the foreground component image in the covered background region, and 
accordingly, description thereof will be omitted. 

[0689] The processing for generating of the predicted image corresponding to the image in the foreground region 
by the mapping unit 807-6 corresponding to Step S806 is the same as the processing described with reference to the 
flowchart shown in Fig. 1 01 except for using the image in the foreground region stored in the foreground region frame 
memory 806 and the coefficient set corresponding to the foreground region, and accordingly, description thereof will 
be omitted. 
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[0690] As described above, the separated image processing unit 1 06 of which configuration is shown in Fig. 92 can 
generate a predicted image for each of images, i.e., the image in the background region, the background component 
image in the uncovered background region, the foreground component image in the uncovered background region, 
the background component image in the covered background region, the foreground component image in the covered 
background region, or the image in the foreground region. 

[0691] Fig. 1 02 is a block diagram which illustrates the configuration of the separated image processing unit 1 06 for 
applying the edge enhancement processing having different effects to each of images, i.e., the image in the background 
region, the background component image in the uncovered background region, the foreground component image in 
the uncovered background region, the background component image in the covered background region, the foreground 
component image in the covered background region, or the image in the foreground region. 
[0692] Background region frame memory 901 stores the image in the background region made up of the pixels 
belonging to the background region supplied from the foreground/background separation unit 1 05. The background 
region frame memory 901 supplies the stored image in the background region to an edge enhancing unit 907-1 . 
[0693] Uncovered background region background component image frame memory 902 stores the background com- 
ponent image in the uncovered background region supplied from the foreground/background separation unit 105. The 
uncovered background region background component image frame memory 902 supplies the stored background com- 
ponent image in the uncovered background region to an edge enhancing unit 907-2. 

[0694] Uncovered background region foreground component image frame memory 903 stores the foreground com- 
ponent image in the uncovered background region supplied from the foreground/background separation unit 105. The 
uncovered background region foreground component image frame memory 903 supplies the stored foreground com- 
ponent image in the uncovered background region to an edge enhancing unit 907-3. 

[0695] Covered background region background component image frame memory 904 stores the background com : 
ponent image in the covered background region supplied from the foreground/background separation unit 105. The 
covered background region background component image frame memory 904 supplies the stored background com- 
ponent image In the covered background region to an edge enhancing unit 907-4. 

[0696] Covered background region foreground component image frame memory 905 stores the foreground compo- 
nent image in the covered background region supplied from the foreground/background separation unit 105. The cov- 
ered background region foreground component image frame memory 905 supplies the stored foreground component 
image in the covered background region to an edge enhancing unit 907-5. 

[0697] Foreground region frame memory 906 stores the image in the foreground region made up of the pixels be- 
longing to the foreground region supplied from the foreground/background separation unit 1 05. The foreground region 
frame memory 906 supplies the stored image in the foreground region to an edge enhancing unit 907-6. 
[0698] The edge enhancing unit 907-1 supplies the image in the background region, which has been subjected to 
edge enhancement by applying the edge enhancement processing suitable for the image in the background region, 
to the image in the background region stored in the background region frame memory 901 , to a synthesizing unit 908. 
[0699] For example, the edge enhancing unit 907-1 performs the edge enhancement processing which further en- 
hances edges for the image in the background region which is the still image, as compared with the foreground region. 
Thus the sense-of- resolution of the image in the background region can be improved without unnatural degradation 
of the image occurring in the event of applying the processing of edge enhancement to a moving image. 
[0700] The edge enhancing unit 907-2 applies the-edge enhancement processing suitable for the background com- 
ponent image in the uncovered background region, to the image stored in the uncovered background region background 
component image frame memory 902, and supplies the image which has been subjected to edge enhancement to the 
synthesizing unit 908. 

[0701] For example, the edge enhancing unit 907-2 performs the edge enhancement processing which further en- 
hances edges for the background component image in the uncovered background region which is the still image, as 
compared with the foreground region. Thus the sense-of-resolution of the image in the background region can be 
improved without unnatural degradation of the image occurring in the event of applying the processing of edge en- 
hancement to a moving image. 

[0702] The edge enhancing unit 907-3 applies edge enhancement processing suitable to the foreground component 
image of the uncovered background region, to the image stored in the uncovered background region foreground com- 
ponent image frame memory 903, and supplies the image which has been subjected to edge enhancement to the 
synthesizing unit 908. 

[0703] For example, the edge enhancing unit 907-3 performs the processing of edge enhancement, which enhances 
the edge for the foreground component image of the covered background region made up of moving foreground com- 
ponents less than as compared with the background region. Thus, the sense-of-resolution of the foreground component 
image of the covered background region can be improved without unnatural degradation of the image occurring in the 
event of applying the processing of edge enhancement to a moving image. 

[0704] The edge enhancing unit 907-4 applies edge enhancement processing suitable to the background component 
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image of the covered background region, to the image stored in the covered background region background component 
image frame memory 904. and supplies the image which has been subjected to edge enhancement to the synthesizing 
unit 908. 

[0705] For example, the edge enhancing unit 907-4 performs the processing of edge enhancement, which further 
enhances the edge for the background component image of the covered background region which is a still image, as 
compared with the foreground region. Thus, the sense-of-resolution of the background region image can be improved 
without unnatural degradation of the image occurring in the event of applying the processing of edge enhancement to 
a moving image. 

[0706] The edge enhancing unit 907-5 applies edge enhancement processing suitable to the foreground component 
image of the covered background region, to the image stored in the covered background region foreground component 
image frame memory 905, and supplies the image which has been subjected to edge enhancement to the synthesizing 
unit 908. 

[0707] For example, the edge enhancing unit 907-5 performs the processing of edge enhancement, which enhances 
the edge for the foreground component image of the covered background region made up of moving foreground com- 
ponents less than as compared with the background region. Thus, the sense-of-resolution of the foreground region 
image of the covered background region can be improved without unnatural degradation of the image occurring in the 
event of applying the processing of edge enhancement to a moving image. 

[0708] The edge enhancing unit 907-6 applies edge enhancement processing suitable to the foreground region im- 
age, to the foreground region image stored in the foreground region frame memory 906, and supplies the foreground 
region image which has been subjected to edge enhancement to the synthesizing unit 908. 

[0709] For example, the edge enhancing unit 907-6 performs the processing of edge enhancement, which enhances 
the edge of the moving foreground region image less than as compared with the background region. Thus, the sense- 
of-resolution of the foreground region image can be improved without unnatural degradation of the image occurring in 
the event of applying the processing of edge enhancement to a moving image. 

[0710] The synthesizing unit 908 synthesizes the background region image subjected to edge enhancing that has 
been supplied from the edge enhancing unit 907-1 , the background component image of the uncovered background 
region subjected to edge enhancing that has been supplied from the edge enhancing unit 907-2, the foreground com- 
ponent image of the uncovered background region subjected to edge enhancing that has been supplied from the edge 
enhancing unit 907-3, the background component image of the covered background region subjected to edge enhanc- 
ing that has been supplied from the edge enhancing unit 907-4, the foreground component image of the covered 
background region subjected to edge enhancing that has been supplied from the edge enhancing unit 907-5, and the 
foreground region image subjected to edge enhancing that has been supplied from the edge enhancing unit 907-6, 
and supplies the synthesized image to the frame memory 909. 

[071 1 ] The frame memory 909 stores the synthesized image supplied from the synthesizing unit 908, and also outputs 
the stored image as an output image. 

[0712] Thus, the separated image processing unit 106 of which configuration is shown in Fig. 74 applies edge en- 
hancement processing corresponding to the nature of each image of each of regions, i.e., the background region, 
uncovered background region, covered background region, or the foreground region, so the sense-of-resolution of the 
image can be improved without unnatural degradation of the image occurring. 

[0713] In the event that there is no need to differentiate the edge enhancing units 907-1 through 907-6, these will 
be simply referred to as the edge enhancing unit 907. 

[071 4] Fig. 1 03 is a block diagram which illustrates the configuration of the edge enhancing unit 907. The separated 
input image is input to a high pass filter 921 and an addition unit 923. 

[0715] The high pass filter 921 extracts the components wherein the pixel value changes drastically with regard to 
pixel position, i.e., the high image frequency components from the input image based upon the input filter coefficients, 
and removes the components wherein the change of the pixel value is small with regard to the pixel position, i.e., the 
low image frequency components, and generates an edge image. 

[0716] For example, in the event of inputting the image shown in Fig. 104A, the high pass filter 921 generates the 
edge image shown in Fig. 104B. 

[0717] In the event that the input filter coefficients change, the high pass filter 921 changes the image frequencies 
which are to be extracted, the image frequencies which are to be removed, and the gain for the image which is to be 
extracted. 

[0718] Referring to Fig. 105 through Fig. 108, the relationship between the filter coefficients and the edge image will 
be described. 

[0719] Fig. 105 is a diagram which illustrates the first example of the filter coefficients. In Fig. 105, E indicates the 
exponent of 10. For example, E-04 indicates 10* 4 , and E-02 indicates 10* 2 . 

[0720] For example, the high pass filter 92 1 multiplies each of pixel values, i.e., the pixel value of the pixel of interest, 
the pixel values of the pixels distanced from the pixel of interest by 1 pixel to 15 pixels in a predetermined direction in 
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the spatial direction Y, and the pixel values of pixels distanced from the pixel of interest by 1 pixel to 1 5 pixels in another 
direction in the spatial direction Y, by the corresponding coefficient of the filter coefficients shown in Fig. 1 05. The high 
pass filter 921 calculates the sum of the results obtained by multiplying each pixel value of the pixels by the coefficient 
corresponding thereto, and sets the calculated sum for the pixel value of the pixel of interest. 
s [0721] For example, in the event of using the filter coefficients shown in Fig. 1 05, the high pass filter 921 multiplies 
the pixel value of the pixel of interest by 1.2169396, multiplies the pixel value of the pixel distanced from the pixel of 
interest by 1 pixel in the upper direction in the screen by -0.52530356, and multiplies the pixel value of the pixel distanced 
from the pixel of interest by 2 pixels in the upper direction in the screen by -0.22739914. 

[0722] In the same way, in the event of using the filter coefficients shown in Fig. 1 05, the high pass filter 921 multiplies 
io each of pixels distanced from the pixel of interest by 3 pixels to 13 pixels in the upper direction in the screen by the 
corresponding coefficient, multiplies the pixel value of the pixel distanced from the pixel of interest by 14 pixels in the 
upper direction in the screen by -0.00022540586, and multiplies the pixel value of the pixel distanced from the pixel of 
interest by 15 pixels in the upper direction in the screen by -0.00039273163. 

[0723] In the event of using the filter coefficients shown in Fig. 1 05, in the same way, the high pass filter 921 multiplies 
15 each of pixels distanced from the pixel of interest by 1 pixel to 15 pixels in the bottom direction in the screen by the 
corresponding coefficient. 

[0724] The high pass filter 921 calculates the sum of results obtained by multiplying the pixel value of the pixel of 
interest, each pixel value of pixels distanced from the pixel of interest by 1 pixel to 15 pixels in the top direction in the 
screen, and each pixel value of pixels distanced from the pixel of interest by 1 pixel to 15 pixels in the bottom direction 
20 in the screen, by the corresponding coefficient. The high pass filter 921 sets the calculated sum to the pixel value of 
the pixel of interest. 

[0725] The high pass filter 921 moves the position of the pixel of interest in sequence in the spatial direction X, 
repeats the above-described processing, and calculates pixel values for the entire screen. 

[0726] The high pass filter 921 then multiplies the pixel value of the interest, each pixel value of pixels distanced 
25 from the pixel of interest by 1 pixel to 15 pixels in a predetermined direction in the spatial direction X, and each pixel 
value of pixels distanced from the pixel of interest by 1 pixel to 15 pixels in another direction in the spatial direction X, 
in the image of which pixel values are calculated based upon the coefficients described above, by the corresponding 
coefficient of the filter coefficients shown in Fig. 1 05. The high pass filter 921 calculates the sum of the results obtained 
by multiplying each pixel value of pixels by the corresponding coefficient, and sets the calculated sum to the pixel value 
30 of the pixel of interest. 

[0727] The high pass filter 921 moves the position of the pixel of interest in sequence in the spatial direction Y, repeats 
the above-described processing, and calculates pixel values of pixels for the entire image. 

[0728] That is to say, in this case, the high pass filter 921 is a so-called one-dimensional filter using the coefficients 
shown in Fig. 1 05. 

35 [0729] Fig. 106 is a diagram which illustrates the operation of the high pass filter 921 in the event of using the 

coefficients shown in Fig. 1 05. As shown in Fig. 1 06, the maximum gain for the extracted image component at the high 

pass filter 921 is 1 in the event of using the coefficients shown in Fig. 105. 

[0730] Fig. 1 07 is a diagram which illustrates the second example of the filter coefficients. 

[0731] Fig. 108 is a diagram which illustrates the operation of the high pass filter 921 in the event that the same 
40 processing as the processing using the filter coefficients shown in Fig. 1 05, is performed using the coefficients shown 

in Fig. 107. As shown in Fig. 108, in the event of using the coefficients shown in Fig. 107, the maximum gain for 

extracted image component at the high pass filter 921 Is 1 .5. 

[0732] As described above, the high pass filter 921 changes the gain for the extracted image component by the 
supplied filter coefficients. 

<5 [0733] While examples are not shown here, in the event of supplying different filter coefficients, the high pass filter 
921 can change the extracted image frequencies and the removed image frequencies in the same way. 
[0734] Returning to Fig. 1 03, the high pass filter 921 supplies the generated edge image to a gain adjustment unit 922 . 
[0735] The gain adjustment unit 922 amplifies or decays the edge image supplied "from the high pass filter 921 based 
upon the input gain adjustment coefficients. In the event that the input gain adjustment coefficient is altered, the gain 

50 adjustment unit 922 changes the amplification ratio {or decay ratio) of the edge image. For example, in the event of 
inputting the gain adjustment coefficients designating an amplification ratio which is equal to or more than 1 , the gain 
adjustment unit 922 amplifies the edge image, and in the event of inputting the gain adjustment coefficients designating 
the amplification ratio which is less than 1 , the gain adjustment unit 922 decays the edge image. 
[0736] The gain adjustment unit 922 supplies the edge image, which has been subjected to gain adjustment, to the 

55 addition unit 923. 

[0737] The addition unit 923 adds the divided input image and the edge image which has been subjected to gain 
adjustment supplied from the gain adjustment unit 922, and outputs the added image. 

[0738] For example, in the event of inputting the Input image shown in Fig. 1 04A, and supplying the edge image 
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shown in Fig. 104B from the high pass filter 921, the addition unit 923 adds the input image shown in Fig. 104A and 
the edge image shown in Fig. 104B, and outputs the image shown in Fig. 104C. 

[0739] As described above, the edge enhancing unit 907 applies the edge enhancement processing for the divided 
image. 

[0740] For example, the edge enhancing unit 907-1 of which configuration is shown in Fig. 103 applies the edge 
enhancement processing of which degree is even higher, to the image in the background region using the coefficients 
shown in Fig. 107. The edge enhancing unit 907-6 of which configuration is shown in Fig. 103 applies the edge en- 
hancement processing of which degree is relatively lower, to the image in the foreground region using the coefficients 
shown in Fig. 105 

[0741] Fig. 109 is a block diagram which illustrates another configuration of the edge enhancing unit 907. In the 
example shown in Fig. 109, the edge enhancing unit 907 comprises a filter 941 . 

[0742] The filter 941 generates an edge enhancement image by amplifying the components wherein the pixel value 
changes drastically with regard to the pixel position, i.e., the high image frequency components in the input image, 
based upon the input filter coefficients. 

[0743] For example, in the event of supplying the coefficients shown by way of an example in Fig. 1 1 0, the filter 941 
performs the same processing as the processing described with regard to the high pass filter 921, based upon the 
coefficients shown by way of an example in Fig. 110. 

[0744] Fig. 1 1 1 is a diagram which illustrates the operation of the filter 941 in the event of using the coefficients shown 
in Fig. 1 1 0. As described in Fig. 1 1 1 , in the event of using the coefficients shown in Fig. 1 1 0, the filter 941 amplifies the 
high image frequency components to double, passes the low image frequency components as they are, and generates 
an edge enhancement image. 

[0745] In the event of using the coefficients shown in Fig. 110, the filter 941 outputs the same output image as the 
output image from the edge enhancing unit 907 of which configuration is shown in Fig. 1 03 wherein that the coefficients 
shown in Fig. 105 are used and the gain at the gain adjustment unit 922 is 1 . 

[0746] Fig. 1 1 2 is a diagram which illustrates the second example of the filter coefficients supplied to the filter 941 . 
[0747] Fig. 1 1 3 is a diagram which illustrates the operation of the filter 941 in the event of using the coefficients shown 
in Fig. 112. As shown in Fig. 113, in the event of using the coefficients shown in Fig. 112, the filter 941 amplifies the 
high image frequency components to 2.5 times, allows the low image frequency components to pass as they are, and 
generates an edge enhancement image. 

[0748] In the event of using the coefficients shown in Fig. 112, the filter 941 outputs the same output image as the 
output image from the edge enhancing unit 907 of which configuration is shown in Fig. 103 in the event that the coef- 
ficients shown in Fig. 107 are used and the gain of the gain adjustment unit 922 is 1 . 

[0749] As described above, the edge enhancing unit 907 of which configuration is shown in Fig. 1 09 can change the 
degree of edge enhancement in the image by altering the gain of the high frequency components in the image, by the 
input filter coefficients. 

[0750] For example, the edge enhancing unit 907-1 of which configuration is shown in Fig. 109 applies the edge 
enhancement processing of which degree is even higher, using the coefficients shown in Fig. 112, to the image in the 
background region. The edge enhancing unit 907-6 of which configuration is shown in Fig. 109 applies the edge en- 
hancement processing of which degree is relatively tower, using the coefficients shown in Fig. 1 10, to the image in the 
foreground region. 

[0751] As described above, the edge enhancing unit 907-1 through the edge enhancing unit 907-6 perform edge 
enhancement processing corresponding to the nature of the divided image based upon different filter coefficients or 
different gain adjustment coefficients, for example. 

[0752] Fig. 114 is a diagram which describes the processing of the separated image processing unit 106 of which 
configuration is shown in Fig. 102. 

[0753] The foreground region, uncovered background region, covered background region, and background region, 
in the input image are specified by the region specifying unit 103. The input image wherein regions are specified is 
separated into the image in the background region, the background component image in the uncovered background 
region, the foreground component image in the uncovered background region, the background component image in 
the covered background region, the foreground component image in the covered background region, and the image 
in the foreground region, by the foreground/background separation unit 105. 

[0754] The separated image processing unit 1 06 of which configuration is shown in Fig. 1 02 performs edge enhance- 
ment processing for each of the image in the background region, the background component image in the uncovered 
background region, the foreground component image in the uncovered background region, the background component 
image in the covered background region, the foreground component image in the covered background region, and the 
image in the foreground region, corresponding to the nature of each image. 

[0755] The image in the background region, the background component image in the uncovered background region, 
the foreground component image in the uncovered background region, the background component image in the cov- 
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ered background region, the foreground component image in the covered background region, and the image in the 
foreground region, each of which has been subjected to edge enhancement, are synthesized. 
[0756] Referring to the flowchart shown in Fig. 115, the processing for edge enhancement by the separated image 
processing unit 106 of which configuration is shown in Fig. 102 will now be described. 
5 [0757] In Step S901 , the edge enhancing unit 907-1 performs edge enhancement of the background region image 
stored in the background region frame memory 901 , by edge enhancement processing corresponding to the nature of 
the background region image. 

[0758] In Step S902, the edge enhancing unit 907-2 performs edge enhancement of the background component 
image of the uncovered background region, stored in the uncovered background region background component image 
10 frame memory 902, by edge enhancement processing corresponding to the nature of the background component 
image of the uncovered background region. 

[0759] In Step S903, the edge enhancing unit 907-3 performs edge enhancement of the foreground component 
image of the uncovered background region, stored in the uncovered background region foreground component image 
frame memory 903, by edge enhancement processing corresponding to the nature of the foreground component image 

'5 of the uncovered background region. 

[0760] In Step S904, the edge enhancing unit 907-4 performs edge enhancement of the background component 
image of the covered background region, stored in the covered background region background component image 
frame memory 904, by edge enhancement processing corresponding to the nature of the background component 
image of the covered background region. 

20 [0761] In Step S905, the edge enhancing unit 907-5 performs edge enhancement of the foreground component 
image of the covered background region, stored in the covered background region foreground component image frame 
memory 905, by edge enhancement processing corresponding to the nature of the foreground component image of 
the covered background region. 

[0762] In Step S906, the edge enhancing unit 907-6 performs edge enhancement of the foreground region image 
25 stored in the foreground region frame memory 906, by edge enhancement processing corresponding to the nature of 
the foreground region image. 

[0763] In Step S907, the synthesizing unit 908 synthesizes the foreground region image, background region image, 
foreground component image of the covered background region, background component image of the covered back- 
ground region, foreground component image of the uncovered background region, and background component image 
30 of the uncovered background region, regarding which each has been subjected to edge enhancement. The synthesizing 
unit 908 supplies the synthesized image to the frame memory 909. The frame memory 909 stores the image supplied 
from the synthesizing unit 908. 

[0764] In Step S908, the frame memory 909 outputs the synthesized image stored therein, and the processing ends. 
[0765] Thus, the separated image processing unit 106 of which the configuration shown in Fig. 102 can execute 
35 edge enhancement processing corresponding to the nature of each of the foreground region image, background region 
image, foreground component image of the covered background region, background component image of the covered 
background region, foreground component image of the uncovered background region, and background component 
image of the uncovered background region, so the sense-of-reso!ution can be improved without causing unnatural 
distortion in moving images. 

40 [0766] Note that it is needless to say that the processing in Step S901 through Step S906 can be performed in a 
serial manner or in a parallel manner. 

[0767] Also, the processing performed by the separated image processing unit 1 06 is not restricted to the generating 
of the coefficients corresponding to an SD image and an HD image, or the processing for generating of an HD image 
from an SD image, an arrangement may be made wherein an even higher resolution image in the spatial direction is 
45 generated by generating the coefficients for generating an even higher resolution image in the spatial direction, for 
example. Moreover, an arrangement may be made wherein the separated image processing unit 106 performs the 
processing for generating an even higher resolution image in the time direction. 

[0768] Note that an arrangement may be made wherein the separated image processing unit 106 performs other 
processing, e.g., image size conversion into a desired size, extracting of color signals such as RGB, removal of noise, 

50 image compression, encoding, or the like, as well as the processing for creation of resolution by the class classification 
adaptation processing, or edge enhancement processing, for each image of the specified region. For example, the 
compression ratio can be increased with little deterioration of the image over conventional arrangements by the sep- 
arated image processing unit 1 06 compressing images of each of the regions with low compression ratio in directions 
following movement vectors and high compression ratio in directions orthogonal to movement vectors, based on move- 

55 ment vectors corresponding to images of each of the regions. 

[0769] Fig. 1 1 6 is a block diagram illustrating another configuration of the functions of the image processing device 
for separating an input image and processing each separated image. While the image processing device shown in Fig. 
11 performs region specification and calculation of the mixture ratio a serially, the image processing device shown in 
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Fig. 116 performs region specification and calculation of the mixture ratio a in parallel. 

[0770] The same portions as the functions shown in the block diagram in Fig. 11 are denoted by the same reference 
numerals, and description thereof will be omitted. 

[0771] The input image is supplied to the object extracting unit 101 , region specifying unit 103, mixture ratio calcu- 
lating unit 1101, and foreground/background separation unit 1102. 

[0772] Based on an input image, the mixture ratio calculating unit 1101 calculates an estimated mixture ratio in a 
case wherein a pixel is assumed to belong to the covered background region, and an estimated mixture ratio in a case 
wherein the pixel is assumed to belong to the uncovered background region, for each of the pixels contained in the 
input image, and supplies the estimated mixture ratio in a case wherein the pixel is assumed to belong to the covered 
background region and the estimated mixture ratio in a case wherein the pixel is assumed to belong to the uncovered 
background region, thus calculated, to the foreground/background separation unit 1102. 

[0773] Fig. 11 7 is a block diagram which illustrates one example of the configuration of the mixture ratio calculation 
unit 1101. 

[0774] The estimated mixture ratio processing unit 401 shown in Fig. 1 1 7 is the same as the estimated mixture ratio 
processing unit 401 shown in Fig. 59. The estimated mixture ratio processing unit 402 shown in Fig. 117 is the same 
as the estimated mixture ratio processing unit 402 shown in Fig. 59. 

[0775] The estimated mixture ratio processing unit 401 calculates the estimated mixture ratio for each pixel by the 
computation corresponding to the model of the covered background region based upon the input image, and outputs 
the calculated estimated mixture ratio. 

[0776] The estimated mixture ratio processing unit 402 calculates the estimated mixture ratio for each pixel by the 
computation corresponding to the model of the uncovered background region based upon the input image, and outputs 
the calculated estimated mixture ratio. 

[0777] Based on the estimated mixture ratio in a case wherein the pixel is assumed to belong to the covered back- 
ground region and the estimated mixture ratio in a case wherein the pixel is assumed to belong to the uncovered 
background region, supplied from the mixture ratio calculating unit 1101 , and the region information supplied from the 
region specifying unit 1 03, the foreground/background separation unit 1 1 02 separates the input image into a foreground 
region image, background region image, foreground component image of the covered background region, background 
component image of the covered background region, foreground component image of the uncovered background re- 
gion, and background component image of the uncovered background region, and supplies the separated images to 
the separated image processing unit 106. 

[0778] Fig. 1 1 8 is a block diagram which illustrates one example of the configuration of the foreground/background 
separation unit 1102. 

[0779] The same portions as the foreground/background separation unit 1 05 shown in Fig. 77 are denoted by the 
same reference numerals, and description thereof will be omitted. 

[0780] A selection unit 1121 selects either of the estimated mixture ratio wherein an assumption is made that the 
pixel belongs to the covered background region, or the estimated mixture ratio wherein an assumption is made that 
the pixel belongs to the uncovered background region, supplied from the mixture ratio calculation unit 1 1 01 based upon 
the region information supplied from the region specifying unit 103, and supplies the selected estimated mixture ratio 
as a mixture ratio a to the separation unit 601 . 

[0781 ] The separation unit 601 extracts the foreground components and the background components from the pixel 
values of the pixels belonging to the mixed region based upon the mixture ratio a supplied from the selection unit 1121 
and the region information, and separates into the background component image in the uncovered background region, 
the foreground component image in the uncovered background region, the background component image in the cov- 
ered background region, and the foreground component image in the covered background region. 
[0782] The configuration of the separation unit 601 may be the same as the configuration shown in Fig. 82. 
[0783] As described above, the image processing device of which configuration is shown in Fig. 116 can perform 
processing for each of images, i.e., the image in the background region, the background component image in the 
uncovered background region, the foreground component image in the uncovered background region, the background 
component image in the covered background region, the foreground component image in the covered background 
region, and the image in the foreground region, corresponding to the nature of each image. 

[0784] As described above, the image processing device of the present invention separates the input image into the 
image in the background region, the background component image in the uncovered background region, the foreground 
component image in the uncovered background region, the background component image in the covered background 
region, the foreground component image in the covered background region, and the image in the foreground region, 
and performs processing suitable to each of separated images, and accordingly, generates an even higher resolution 
image, for example. 

[0785] Fig. 11 9 is a block diagram which illustrates another configuration of the image processing device. 

[0786] The same portions as shown in Fig. 1 1 are denoted by the same reference numerals, and description thereof 
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will be omitted. 

[0787] The input image supplied to the image processing device is supplied to the object extracting unit 101 , the 
region specifying unit 1 03, the mixture ratio calculation unit 104, and the foreground/background separation unit 2001 . 
[0788] The foreground/background separating unit 2001 separates the input images into foreground component 
images which consist of only the foreground components corresponding to the foreground object and background 
component images which consist of only the background components based upon the region information supplied from 
the region specifying unit 1 03 and the mixture ratio a supplied from the mixture ratio calculating unit 1 04, supplies the 
foreground component image to the movement blurring removal unit 2002, and supplies the background component 
image to a correction unit 2003. 

[0789] The movement blurring removal unit 2002 decides the increment of processing, which indicates one or more 
pixels included in the foreground component images, based upon the movement amount v which is led from the move- 
ment vector, and the region information. An increment of processing is the data which designates one group of the 
pixels which are the object for adjustment processing for the movement blurring amount. 

[0790] The movement blurring removal unit 2002 removes movement blurring contained in the foreground component 
image based upon the foreground component image supplied from the foreground/background separation unit 2001 , 
the movement vector and the position information thereof supplied from the movement detecting unit 102, and the 
processing increment, and outputs the foreground component image which has been subjected to removal of movement 
blurring, to a movement-blurring-removed-image processing unit 2004. 

[0791] The correction unit 2003 corrects the pixel value of a pixel corresponding to the mixed region in the background 
component image. The pixel value of a pixel corresponding to the mixed region in the background component image 
is calculated by subtracting the foreground component from the pixel value of a pixel in the mixed region prior to 
separation. Accordingly, the pixel value of a pixel corresponding to the mixed region in the background component 
image decreases corresponding to the mixture ratio a, as compared to the pixel value of a pixel in the adjacent back- 
ground region. 

[0792] The correction unit 2003 corrects the decrease of the gain corresponding to the mixture ratio a of the pixel 
value of a pixel corresponding to the mixed region in the background component image, as described above, and 
supplies the corrected background component image to the movement-blurring-removed-image processing unit 2004. 
[0793] The movement-blurring-removed-image processing unit 2004 individually performs processing for the fore- 
ground component image which has been subjected to removal of movement blurring and the corrected background 
component image. 

[0794] For example, the movement-biurring-removed-image processing unit 2004 generates coefficients which are 
used in the classifying adaptation processing for generating an even higher resolution image, for each foreground 
component image which has been subjected to removal of movement blurring, and for each corrected background 
component image. 

[0795] For example, the movement-blurring-removed-image processing unit 2004 creates an even higher resolution 
image by applying the classifying adaptation processing to each foreground component image which has been sub- 
jected to removal of movement biurring, and for each corrected background component image. 
[0796] Also, for example, the movement-blurring-removed-image processing unit 2004 applies edge enhancement 
processing of which degree is different using different coefficients for each of images, i.e., the foreground component 
image subjected to removal of movement blurring, and the background component image subjected to correction. 
[0797] Fig. 120 is a diagram which illustrates the correspondence of the image divided into pixels each of which 
belongs to the foreground region, background region, covered background region, or uncovered background region, 
to a model diagram wherein the pixel values of pixels develop over the time direction. 

[0798] As shown in Fig. 120, the region specifying unit 103 specifies the foreground region, background region, 
covered background region, and uncovered background region, of the input image. 

[0799] As shown in Fig. 121 , the separated background component image is corrected for the pixel values of the 
mixed region, and the separated foreground component image is subjected to removal of movement blurring. 
[0800] As shown in Fig. 122, the input image is divided into regions, and is separated into foreground components 
and background components. The separated input image is synthesized into the background component image and 
the foreground component image. Movement blurring contained in the foreground component image is removed. The 
background component image is corrected with regard to the pixel values corresponding to the mixed region. 
[0801 ] The foreground component image subjected to removal of movement blurring and the background component 
image subjected to correction are individually processed. 

[0802] Fig. 123 is a flowchart which describes the image processing of the image processing device according to 
the present invention. 

[0803] In Step S2001, the region specifying unit 103 specifies the foreground region, background region, covered 
background region, and uncovered background region, in the input image based upon the movement vector and the 
position information thereof supplied from the movement detecting unit 102 and the input image. The processing of 
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region specification in Step S2001 is the same as the processing shown in Step S101, so detailed description of the 
processing will be omitted. 

[0804] In Step S2002, the mixture ratio calculation unit 104 calculates the mixture ratio a based upon the region 
information supplied from the region specifying unit 103 and the input image. The processing for calculation of the 
mixture ratio a in Step S2002 is the same as the processing in Step S102, so detailed description of the processing 
will be omitted. 

[0805] in Step S2003, the foreground/background separation unit 2001 separates the input image into the image in 
theforeground region, the image in the background region, the foreground component image in the covered background 
region, the background component image in the covered background region, the foreground component image in the 
uncovered background region, and the background component image in the uncovered background region, based 
upon the region information supplied from the region specifying unit 103 and the mixture ratio a supplied from the 
mixture ratio calculation unit 1 04. Details of the processing for separation of the image by the foreground/background 
separation unit 2001 will be described later. 

[0806] In Step S2004, the movement blurring removal unit 2002 removes movement blurring from the foreground 
component image supplied from the foreground/background separation unit 2001 , based upon the movement vector 
and the position information thereof supplied from the movement detecting unit 1 02 and the region information supplied 
from the region specifying unit 1 03. 

[0807] Details of the processing for removal of movement blurring by the movement blurring removal unit 2002 will 
be described later. 

[0808] In Step S2005, the correction unit 2003 corrects the pixel values corresponding to the mixed region of the 
background component image supplied from the foreground/background separation unit 2001 . 
[0809] In Step S2006, the movement-bturring-removed-lmage processing unit 2004 performs image processing for 
each foreground component image which has been subjected to removal of movement blurring and each background 
component image which has been corrected, and the processing ends. Details of the image processing performed by 
the movement-blurring-removed-image processing unit 2004 will be described later. 

[0810] As described above, the image processing device according to the present invention separates the input 
image into the foreground component image and the background component image, removes movement blurring from 
the foreground component image, and performs image processing for each of the foreground component image sub- 
jected to removal of movement blurring, and the background component image. 

[0811] A description will now be made with regard to the foreground/background separation unit 2001 . Fig. 124 is a 
block diagram which illustrates an example of the configuration of the foreground/background separation unit 2001 . 
The input image supplied to the foreground/background separation unit 2001 is supplied to a separation unit 2601 , a 
switch 2602, and a switch 2604. The region information supplied from the region specifying unit 103, which indicates 
the covered background region and the uncovered background region, is supplied to the separation unit 2601 . The 
region information which indicates the foreground region is supplied to the switch 2602. The region information which 
indicates the background region is supplied to the switch 2604. 

[0812] The mixture ratio a supplied from the mixture ratio calculation unit 1 04 is supplied to the separation unit 2601 . 
[0813] The separation unit 2601 separates the foreground components from the input image based upon the region 
information indicating the covered background region, the region information indicating the uncovered background 
region, and the mixture ratio a, and supplies the separated foreground components to a synthesizing unit 2603, as 
well as separating the background components from the input image, and supplying the separated background com- 
ponents to the synthesizing unit 2605. 

[0814] In the event that the pixel corresponding to the foreground is input, the switch 2602 is closed based upon the 
region information indicating the foreground region, and supplies only the pixels corresponding to the foreground in- 
cluded in the input image to the synthesizing unit 2603. 

[0815] In the event that the pixel corresponding to the background is input, the switch 2604 is closed based upon 
the region information indicating the background region, and supplies only the pixels corresponding to the background 
included in the input image to the synthesizing unit 2605. 

[0816] The synthesizing unit 2603 synthesizes the foreground component image based upon the components cor- 
responding to the foreground supplied from the separation unit 2601 , and the pixels corresponding to the foreground 
supplied from the switch 2602, and outputs the synthesized foreground component image. Since the foreground region 
and the mixed region are not overlapped, the synthesizing unit 2603 synthesizes the foreground component image, 
for example, by applying the logical sum computation to the components corresponding to the foreground, and the 
pixels corresponding to the foreground. 

[0817] In the initialization processing which is performed in the first stage of the foreground component image syn- 
thesizing processing, the synthesizing unit 2603 stores the image, wherein all the pixel values are 0, in built-in frame 
memory, and in the foreground component image synthesizing processing, the synthesizing unit 2603 stores (or over- 
writes) the foreground component image. Accordingly, the pixel corresponding to the background region, which is the 
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foreground component image output from the synthesizing unit 2603, stores 0 as a pixel value. 
[081 8] The synthesizing unit 2605 synthesizes the background component image based upon the components cor- 
responding to the background supplied from the separation unit 2601 , and the pixels corresponding to the background 
supplied from the switch 2604, and outputs the synthesized background component image. Since the background 
region and the mixed region are not overlapped, the synthesizing unit 2605 synthesizes the background component 
image, for example, by applying the logical sum computation to the components corresponding to the background, 
and the pixels corresponding to the background. 

[0819] In the initialization processing which is performed in the first stage of the background component image syn- 
thesizing processing, the synthesizing unit 2605 stores the image, wherein ail the pixel values are 0, in built-in frame 
memory, and in the background component image synthesizing processing, the synthesizing unit 2605 stores (or over- 
writes) the background component image. Accordingly, the pixel corresponding to the foreground region, which is the 
background component image output from the synthesizing unit 2605, stores 0 as a pixel value. 
[0820] Fig. 125A and Fig. 125B are diagrams which illustrate the input image input to the foreground/background 
separation unit 2001 , and the foreground component image and the background component image output from the 
foreground/background separation unit 2001. 

[0821] Fig. 125A is a schematic diagram which illustrates the displayed image, and Fig. 125B is a model diagram 
wherein one line of pixels including pixels belonging to the foreground region, pixels belonging to the background 
region, and pixels belonging to the mixed region, corresponding to Fig. 125A, develop over the time direction. 
[0822] As shown in Fig. 125A and Fig. 125B, the background component image output from the foreground/back- 
ground separation unit 2001 is made up of pixels belonging to the background region and background components 
contained in pixels in the mixed region. 

[0823] As shown in Fig. 125A and Fig. 125B, the foreground component image output from the foreground/back- 
ground separation unit 2001 is made up of pixels belonging to the foreground region and foreground components 
contained in pixels in the mixed region. 

[0824] The pixel value of the pixel in the mixed region is separated into the background components and the fore- 
ground components by the foreground/background separation unit 2001 .The separated background components make 
up a background component image along with pixels belonging to the background region. The separated foreground 
components make up a foreground component image along with pixels belonging to the foreground region. 
[0825] As described above, in the foreground component image, the pixel values of the pixels corresponding to the 
background region are set to 0, and the pixels corresponding to the foreground region and the pixels corresponding 
to the mixed region are set to valid pixel values. Similarly in the background component image, the pixel values of the 
pixels corresponding to the foreground region are set to 0, and the pixels corresponding to the background region and 
the pixels corresponding to the mixed region are set to valid pixel values. 

[0826] A description will now be made regarding the separation processing of the foreground components and the 
background components from the pixel belonging to the mixed region performed by the separation unit 2601 . 
[0827] Fig. 126 is a model of an image which indicates two frames of the foreground components and the background 
components, including the foreground corresponding to the object which moves from the left to the right in the drawing. 
In the model of the image shown in Fig. 126, the movement amount v of the foreground is 4, and the virtual dividing 
number is 4. 

[0828] In the frame #n, the left-most pixel and the fourteenth through eighteenth pixels from the left are made up of 
only the background components, and belong to the background region. In the frame #n, the second through fourth 
pixels from the left are made up of the background components and the foreground components, and belong to the 
uncovered background region. In the frame #n, the eleventh through thirteenth pixels from the left are made up of the 
background components and the foreground components, and belong to the covered background region. In the frame 
#n, the fifth through tenth pixels from the left are made up of only the foreground components, and belong to the 
foreground region, 

[0829] In the frame #n + 1 , the first through fifth pixels from the left and the eighteenth pixel from the left are made 
up of only the background components, and belong to the background region. In the frame #n + 1 , the sixth through 
eighth pixels from the left contain the background components and the foreground components, and belong to the 
uncovered background region. In the frame #n + 1 , the fifteenth through seventeenth pixels from the left contain the 
background components and the foreground components, and belong to the covered background region. In the frame 
#n + 1 , the ninth through fourteen pixels from the left are made up of only the foreground components, and belong to 
the foreground region. 

[0830] Fig. 127 is a diagram which describes the processing for separation of the foreground components from the 
pixel belonging to the covered background region, in Fig. 127, a1 through a18 are the mixture ratios corresponding 
to the pixels in the frame #n, respectively. In Fig. 127, the fifteenth through seventeenth pixels from the left belongs to 
the covered background region. 

[0831] The pixel value C15 of the fifteenth pixel from the left in the frame #n is represented in Expression (85). 
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C15 = B15/v + F09/V + F08/v + F07/v 
= al5 • B15 + F09/v + F08/v + F07/v 

« al5 • P15 + F09/V + F08/v + F07/v (85) 

[0832] Here, a15 denotes the mixture ratio of the fifteenth pixel from the left in the frame #n. P15 denotes the pixel 
value of the fifteenth pixel from the left in the frame #n - 1 . 

[0833] The sum 115 of the foreground components of the fifteenth pixel from the left in the frame #n is represented 
in Expression (86) based upon Expression (85). 

f15=FO9/v + F08/v+FO7/v 

= C15-a15- P15 (86) 

[0834] Similarly, the sum f 16 of the foreground components of the sixteenth pixel from the left in the frame #n is 
represented in Expression (87), and the sum f1 7 of the foreground components of the seventeenth pixel from the left 
in the frame #n is represented in Expression (88). 

f16 = C16-ct16. P16 (87) 

f17 = C17-a17 - P17 (88) 

[0835] As described above, the foreground component fc contained in the pixel value C of the pixel belonging to the 
covered background region is calculated by Expression (89). 

fc=C-a-P (89) 
[0836] P denotes the pixel value of the corresponding pixel in the previous frame. 

[0837] Fig. 128 is a diagram which describes the processing for separating the foreground components from the 
pixel belonging to the uncovered background region. In Fig. 1 28 , cx1 through a1 8 denote the mixture ratio corresponding 
to the pixels in the frame #n, respectively. In Fig. 128, the second through fourth pixels from the left belong to the 
uncovered background region. 

[0838] The pixel value C02 of the second pixel from the left in the frame #n is represented in Expression (90). 

C02 = B02/v + B02/\r + B02/v + FOl/v 
■ » <X2 • B02 + FOl/v 
= a2 • N02 + FOl/v (90) 

[0839] Here, a2 denotes the mixture ratio'of the second pixel from the left in the frame #n. N02 denotes the pixel 
value of the second pixel from the left in the frame #n + 1 . 

[0840] The foreground component sum of the second pixel from the left in the frame #n, f02, is represented in Ex- 
pression (91) based upon Expression (90). 

f02 = F01/v 

= C02-a2-N02 (91) 
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[0841 ] Similarly, the foreground component sum of the third pixel from the left in the frame #n, f03, is represented in 
Expression (92), and the foreground component sum of the fourth pixel from the left in the frame #n, f04, is represented 
in Expression (93). 

f03 = C03 - a3 • N03 (92) 



f04 = C04 - a4 . N04 (93) 

[0842] As described above, the foreground component fu contained in the pixel value C of the pixel belonging to the 
uncovered background region is calculated by Expression (94). 

f u = C - a • N (94) 
[0843] N denotes the pixel value of the corresponding pixel in the following frame. 

[0844] As described above, the separation unit 2601 can separate the foreground components and the background 
components from the pixel belonging to the mixed region based upon the information indicating the covered background 
region and the information indicating the uncovered background region, which is included in the region information, 
and the mixture ratio a for each pixel. 

[0845] Fig. 129 is a block diagram which illustrates an example of the configuration of the separation unit 2601 for 
performing the processing described above. The image input to the separation unit 2601 is supplied to frame memory 
2621 , and the region information indicating the covered background region and the uncovered background region and 
the mixture ratio a, supplied from the mixture ratio calculating unit 1 04, are input to a separation processing block 2622. 
[0846] The frame memory 2621 stores the input image in increments of frames. In the event that the frame #n is the 
object of processing, the frame memory 2621 stores the frame #n - 1 which is the frame previous to the frame #n, 
frame #n, and the frame #n + 1 which is the frame following the frame #n. 

[0847] The frame memory 2621 supplies the corresponding pixels in the frame #n - 1 , the frame #n, and the frame 
#n + 1 to the separation processing block 2622. 

[0848] The separation processing block 2622 separates the. foreground components and the background compo- 
nents from the pixel belonging to the mixed region in the frame #n by applying the computation described with reference 
to Fig. 127 and Fig. 128 to the pixel values of corresponding pixels in the frame #n - 1 , the frame #n, and the frame 
#n + 1 , supplied from the frame memory 2621 , based upon the region information indicating the covered background 
region and the uncovered background region, and the mixture ratio a, and supplies to the frame memory 2623. 
[0849] The separation processing block 2622 comprises an uncovered region processing unit 2631 , a covered region 
processing unit 2632, a synthesizing unit 2633, and a synthesizing unit 2634. 

[0850] A multiplication device 2641 of the uncovered region processing unit 2631 multiplies the pixel value of the 
pixel of the frame #n + 1 supplied from the frame memory 2621 by the mixture ratio a, and outputs to a switch 2642. 
In the event that the pixel in the frame #n supplied from the frame memory 2621 (which is corresponding to the pixel 
of the frame #n + 1 ) belongs to the uncovered background region, the switch 2642 is closed, and the pixel value which 
has been multiplied by the mixture ratio a supplied from the multiplication device 2641 is supplied to a computing 
device 2643 and the synthesizing unit 2634. The value wherein the pixel value of the pixel of the frame #n + 1 output 
from the switch 2642 is multiplied by the mixture ratio a is the same as the background component of the pixel value 
of the corresponding pixel in the frame #n. 

[0851] The computing device 2643 obtains the foreground components by subtracting the background components 
supplied from the switch 2642 from the pixel value of the pixel of the frame #n supplied from the frame memory 2621 . 
The computing device 2643 supplies the foreground components of the pixel in the frame #n belonging to the uncovered 
background region, to the synthesizing unit 2633. 

[0852] A multiplication device 2651 of the covered region processing unit 2632 multiplies the pixel value of the pixel 
of the frame #n - 1 supplied from the frame memory 2621 by the mixture ratio a, and outputs to a switch 2652. In the 
event that the pixel in the frame #n supplied from the frame memory 2621 (corresponding to the pixel of the frame #n 
- 1 ) belongs to the covered background region, the switch 2652 is closed, and the pixel value which has been multiplied 
by the mixture ratio a supplied from the multiplication device 2651 is supplied to a computing device 2653 and the 
synthesizing unit 2634. The value wherein the pixel value of the pixel of the frame #n - 1 has been multiplied by the 
mixture ratio a, which is output from the switch 2652, is the same as the background component of the pixel value of 
the corresponding pixel in the frame #n. 

[0853] The computing device 2653 obtains the foreground components by subtracting the background components 
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supplied from the switch 2652 from the pixel value of the pixel of the frame #n supplied from the frame memory 2621 . 
The computing device 2653 supplies the foreground components of the pixel in the frame #n belonging to the covered 
background region, to the synthesizing unit 2633. 

[0854] The synthesizing unit 2633 synthesizes the foreground components of the pixel belonging to the uncovered 
5 background region supplied from the computing device 2643, and the foreground components of the pixel belonging 
to the covered background region supplied from the computing device 2653, in the frame #n, and supplies to the frame 
memory 2623. 

[0855] The synthesizing unit 2634.synthesizes the background components of the pixel belonging to the uncovered 
background region supplied from the switch 2642, and the background components of the pixel belonging to the covered 

io background region supplied from the switch 2652, in the frame #n, and supplies to the frame memory 2623. 

[0856] The frame memory 2623 stores the foreground components and the background components of the pixels 
in the mixed region in the frame #n, supplied from the separation processing block 2622, respectively. 
[0857] The frame memory 2623 outputs the foreground components of the pixels in the mixed region in the frame 
#n stored therein, and the background components of the pixels in the mixed region in the frame #n stored therein. 

is [0858] Using the mixture ratio a which is the feature amount enables complete separation of the foreground com- 
ponents and the background components, contained in the pixel value. 

[0859] The synthesizing unit 2603 generates a foreground component image by synthesizing the foreground com- 
ponents of the pixel in the mixed region in the frame #n output from the separation unit 2601 , and the pixels belonging 
to the foreground region. The synthesizing unit 2605 generates a background component image by synthesizing the 
20 background components of the pixels in the mixed region in the frame #n output from the separation unit 2601 , and 
pixels belonging to the background region. 

[0860] Fig. 130A is a diagram which illustrates an example of the foreground component image corresponding to 
the frame #n shown in Fig. 126. Fig. 130B is a diagram which illustrates an example of the background component 
image corresponding to the frame #n shown in Fig. 1 26. 
25 [0861] Fig. 130A illustrates an example of the foreground component image corresponding to the frame #n shown 
in Fig. 126. Since the left-most pixel and the fourteenth pixel from the left are made up of only the background com- 
ponents before separation of the foreground and the background, the pixel values are 0. 

[0862] The second through fourth pixels from the left belong to the uncovered background region prior to the fore- 
ground and the background being separated, with the background components being 0, and the foreground compo- 
se nents being left as they are. The eleventh through thirteenth pixels belong to the covered background region before 
separation of the foreground and the background, and the background components are 0, and the foreground compo- 
nents are left as they are. Since the fifth through tenth pixels from the left are made up of only the foreground compo- 
nents, those are left as they are. 

[0863] Fig. 130B illustrates an example of the background component image corresponding to the frame #n shown 

35 in Fig. 1 26. The left-most pixel and the fourteenth pixel from the left are made up of only the background components 
prior to the foreground and the background being separated, and accordingly, those are left as they are. 
[0864] The second through fourth pixels from the left belong to the uncovered background region prior to the fore- 
ground and the background being separated, with the foreground components being 0, and the background compo- 
nents being left as they are. The eleventh through thirteenth pixels belong to the covered background region prior to 

40 the foreground and the background being separated, the foreground components being 0, and the background com- 
ponents being left as they are. The fifth through tenth pixels from the left are made up of only the foreground components 
prior to the foreground and the background being separated, and accordingly the pixel values are 0. 
[0865] The separation processing for the foreground and the background by the foreground/background separation 
unit 2001 will now be described, with reference to the flowchart shown In Fig. 131 . In Step S2601 , the frame memory 

45 2621 of the separation unit 2601 obtains the input image, and stores the frame #n which is the object for separation 
of the foreground and the background, as well as the previous frame #n - 1 and the following frame #n + 1 . 
[0866] In Step S2602, the separation processing block 2622 of the separation unit 2601 obtains the region information 
supplied from the mixture ratio calculating unit 104. In Step S2603, the separation processing block 2622 of the sep- 
aration unit 2601 obtains the mixture ratio a supplied from the mixture ratio calculating unit 104. 

so [0867] In Step S2604, the uncovered region processing unit 2631 extracts the background components from the 
pixel value of the pixel belonging to the uncovered background region supplied from the frame memory 2621 based 
upon the region information and the mixture ratio a. 

[0868] In Step S2605, the uncovered region processing unit 2631 extracts the foreground components from the pixel 
value of the pixel belonging to the uncovered background region supplied from the frame memory 2621 based upon 
55 the region information and the mixture ratio a. 

[0869] In Step S2606, the covered region processing unit 2632 extracts the background components from the pixel 
value of the pixel belonging to the covered background region supplied from the frame memory 2621 based upon the 
region information and the mixture ratio a. 
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[0882] The modeling unit 2802 supplies the selected model to an expression generating unit 2803. 
[0883] The expression generating unit 2803 generates a expression based upon a model supplied from the modeling 
unit 2802. The expression generated by the expression generating unit 2803 will be described in a case wherein the 
number of the foreground components is 8, the number of pixels corresponding to the processing increment is 12, the 
movement amount v is 5, and the virtual dividing number is 5, with reference to the model for the foreground component 
image shown in Fig. 134. 

[0884] In the event that the foreground components corresponding to the shutter period/v contained in the foreground 
component image are F01/v through F08/v, the relationships between F01/v through F08/v and the pixel values C01 



through C12 are represented in Expression (95) through Expression (106). 

C01 = F01/V (95) 

C02=F02/v+F01/v (96) 

C03 = F03/V + F02/V + F01 N (97) 

C04 = F04/V + F03/V + F02/v + F01/v (98) 

C05 = F05/V + F04/V + F03/v + F02/V + F01A/ (99) 

C06 = F06/V + F05/V + F04/V + F03/v + F02/v (1 00) 

C07 = F07/V + F06/V + F05/V + F04/V + F03/V (101) 

COB = F08/V + F07/V + F06/V + F05/V + F04/v (1 02) 

C09 = F08/V + F07/V + F06/V + F05/V (1 03) 

C1 0 = F08/V + F07/V + F06/V (1 04) 

C11 = F08/v+F07/v (105) 

C12=F08/V (106) 



[0885] The expression generating unit 2803 generates expressions by transforming the generated expressions. The 
expressions generated by the expression generating unit 2803 are represented in Expression (1 07) through Expression 
(118). 

C01 = 1 • F01/V + 0 • F02/V + 0 • F03/V+ 0 . F04/V+ 0 • 

F05/V+ 0 • F06/V+ 0 . F07/V+ 0 ■ F08/v (1 07) 



C02 = 1 . F01/V + 1 • F02/V + 0 • F03/V+ 0 • F04/V+ 0 . 
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F05/V+ 0 • F05/V+ 0 . F07/V+ 0 - F08/V (1 08) 

C03=1 -F01/V+1 -F02/V+1 . F03/V+ 0 • F04/V+ 0 • 

F05/V+ 0 • F06/V+ 0 • F07/v+ 0 • F08/V (1 09) 

C04 = 1 • F01/V + 1 • F02/V + 1 • F03/V+ 1 ■ F04N+ 0 ■ 

F05/V+ 0 • F06/V+ 0 • F07/V+ 0 • F08/V (1 1 0) 

C05 = 1 . F01/V + 1 • F02/V + 1 • F03/V+ 1 . F04/V+ 1 • 

F05/V+ 0 • F06/V+ 0 • F07/V+ 0 • F08/v (111) 

C06 = 0 • F01/V + 1 • F02/V + 1 • F03/V+ 1 • F04/V+ 1 • 

F05/V+ 1 • F06/V+ 0 • F07/V+ 0 • F08/V (1 1 2) 

C07 = 0 • F01/V + 0 • F02/V + 1 . F03/V+ 1 • F04/V+ 1 • 

F05/V+ 1 • F06/V+ 1 • F07/V+ 0 . F08/V (113) 

C08 = 0 • F01/V + 0 • F02/V + 0 . F03A/+ 1 . F04/V+ 1 • 

F05/V+ 1 • F06/V+ 1 • F07/V+ 1 • F08/v (114) 

C09 = 0 • F01/V + 0 • F02/V + 0 - F03/V+ 0 • F04/V+ 1 • 

F05/V+ 1 • F06/V+ 1 • F07/V+ 1 • F08/V (115) 

C10 = 0 . F01/V + 0 • F02/V + 0 ■ F03/V+ 0 . F04/V+ 0 . 

F05/V+ 1 • F06/V+ 1 • F07A/+ 1 . F08/V (116) 

C11 =0.F01/v + 0.F02/v + 0-F03/v+0.F04/v+0. 

F05/V+ 0 • F06/V+ 1 • F07/V+ 1 • F08/V (117) 

C12 = 0 • F01/V + 0 • F02/V + 0 . F03/V+ 0 - F04/V+ 0 • 

F05/V+ 0 • F06/V+ 0 • F07/V+ 1 ■ F08/V (118) 

[0886] Expression (107) through Expression (118) may be represented as with Expression (119). 
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Cj=]taij-Fi/v (119) 

[0887] In Expression (1 19), j denotes the pixel position. In this example, j has one of the values between 1 and 12. 
Also, i denotes the position of the foreground value. In this example, I has one of the values between 1 and 8. Corre- 
sponding to the values of i and j, aij has one of the values of 0 or 1 . 

[0888] Taking margin of error into consideration, Expression (119) may be represented as with Expression (120). 

08 

Cj = 2,aij-Fi/ v + ej 



(120) 



[0889] In Expression (120), ej denotes the margin of error contained in the pixel of interest, Cj. 
[0890] Expression (120) can be rewritten into Expression (121) 



e J = Cj - 2 aij-Fifv (121) 

i=0J 



[0891] Note that in order to use the least square method, the squared-sum of the margin of error E is defined i 
represented in Expression (122). 



E = Zej 2 (122) 



[0892] To minimize margin of error, the value of the partial derivative of the squared-sum of the margin of error E 
from the variable Fk should become 0. Fk is obtained so as to satisfy Expression (123). 

dFk dFk 

= 2 -2{(Cj -Zaij'Filv)'(-'akjlv) = 0 (123) 

[0893] In Expression (123), the movement amount v is a fixed value, so Expression (124) can be derived. 

22 08 

E akj • (Cj -~Zaij-Fi/v) = 0 (124) 

[0894] Developing Expression (124) and transposing arguments, Expression (125) is obtained. 



12 08 12 

£ (aicj^aij-Fi) = v -Zaicj-Cj (us) 

j~oi i=oi j*0J 
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[0895] Expression (125) is developed into eight expressions, each of which is obtained by substituting one of the 
integers between 1 and 8 for k in Expression (125). The obtained eight expressions may be represented in one ex- 
pression by a matrix. The expression is referred to as a normal equation. 

[0896] An example of the normal equation generated by the expression generating unit 2803 based upon such the 
least square method is represented in Expression (126). 
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[0897] In the event that Expression (126) is represented by A • F = v • C, then C, A, and v are known, and F is 
unknown. Also, while A and v are known at the point of modeling, C becomes known by inputting the pixel value in 
addition operation. 

[0898] The margin of error contained in the pixel C is dispersed by calculating the foreground components by the 
normal equation based upon the least square method. 

[0899] The expression generating unit 2803 supplies the normal equation generated as described above, to the 
addition unit 2804. 

[0900] The addition unit 2804 sets the pixel value C contained in the foreground component image for the expression 
of the matrix supplied from the expression generating unit 2803 based upon the processing increment supplied from 
the processing increment decision unit 2801 . The addition unit 2804 supplies the matrix which the pixel value C is set 
for, to the computing unit 2805. 

[0901] The computing unit 2805 calculates the foreground component Fi/v which has been subjected to removal of 
the movement blurring by the processing based upon the method such as the sweeping method (Gauss-Jordan elim- 
ination), calculates Fi corresponding to one of the integers i between 0 and 8, which is the pixel value of the foreground 
which has been subjected to removal of the movement blurring, and outputs the foreground component image which 
has been subjected to removal of the movement blurring, which is made up of H which is the pixel value which has 
been subjected to removal of the movement blurring as shown by way of an example, shown in Fig. 135. 
[0902] Note that in the foreground component image which has been subjected to removal of the movement blurring 
shown in Fig. 1 35, each of C03 through C1 0 is set to each of F01 through F08 so as not to change the position of the 
foreground component image with regard to the screen, which can correspond to an arbitrary position. 
[0903] Also, as shown in Fig. 136, for example, in the event that the number of pixel corresponding to the processing 
increment is 8 and the movement amount v is 4, the movement blurring removal unit 2002 generates a matrix expression 
represented in Expression (127). 
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(127) 



15 [0904] The movement blurring removal unit 2002 calculates Fi which is the pixel value which has been subjected to 
adjustment of movement blurring by forming expressions of which number corresponds to the length of the processing 
increment. In the same way, in the event that the number of pixel contained in the processing increment is one hundred, 
Fi is calculated by generating expressions corresponding to the one hundred pixels. 

[0905] As described above, the movement blurring removal unit 2002 generates expressions corresponding to the 
20 movement amount v and the processing increment, sets pixel values of the foreground component image for the gen- 
erated expressions, and calculates an foreground component image which has been subjected to removal of movement 
blurring. 

[0906] The processing for removal of movement blurring contained in the foreground component image by the move- 
ment blurring removal unit 2002 will now be descried with reference to the flowchart shown in Fig. 137. 
25 [0907] In Step S2801 , the processing increment decision unit 2801 of the movement blurring removal unit 2002 
generates, the processing increment based upon the movement vector and the region information, and supplies the 
generated processing increment to the modeling unit 2802. 

[0908] In Step S2802, the modeling unit 2802 of the movement blurring removal unit 2002 performs selecting or 
generating of the model corresponding to the movement amount v and the processing increment. In Step S2803, the 

30 expression generating unit 2803 creates the normal equation based upon the selected model. 

[0909] In Step S2804, the addition unit 2804 sets the pixel values of the foreground component image for the created 
normal equation. In Step S2805, the addition unit 2804 judges whether or not the pixel values of all the pixels corre- 
sponding to the processing increment are set, and in the event that judgment is made that not all the pixel values of 
the pixels corresponding to the processing increment have been set, the flow returns to Step S2804 and repeats the 

35 processing of setting the pixel values for the normal equation. 

[0910] In the event that judgment is made that all the pixel values of the pixels of the processing increment have 
been set in Step S2805, the flow proceeds to Step S2806, the computing unit 2805 calculates the pixel values of the 
foreground which has been subjected to removal of movement blurring based upon the normal equation wherein the 
pixel values supplied from the addition unit 2804 are set, and the processing ends. 

40 [0911] As described above, the movement blurring removal unit 2002 can remove movement blurring from the fore- 
ground image containing the movement blurring based upon the movement vector and the region information. 
[0912] That is to say, movement blurring contained in the pixel values which are the sampled data, can be removed. 
[0913] Next, correction of the background component image will be described. 

[0914] Fig. 1 38 is a diagram which illustrates an example of the model of the background component image corre- 
45 spending to the model of the foreground component image shown by way of an example shown in Fig. 134. 

[0915] As shown in Fig. 138, the foreground components have been removed, so pixel values of the pixels of the 
background component image corresponding to the mixed region in the original input image are made up of a small 
number of background components as compared with the pixels corresponding to the background region in the original 
input image, corresponding to the mixture ratio a. 
so [0916] For example, in the background component image shown by way of an example shown in Fig. 138, the pixel 
value C01 is made up of four background components B02/Vs, the pixel value C02 is made up of three background 
components B03/Vs, the pixel value C03 is made up of two background components B04/Vs, and the pixel value C04 
is made up of one background component B05W. 

[0917] Also, with the background component image shown byway of an example shown in Fig. 138, the pixel value 
55 C09 is made up of one background component B1 0/V, thee pixel value C1 0 is made up of two background components 
B11A/S, the pixel value C11 is made up of three background components B12/Vs, and the pixel value C12 is made up 
of four background components B13A/S. 

[0918] As described above, the pixel value of a pixel corresponding to the mixed region in the original input image 
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is made up of a small number of background components as compared with the pixel corresponding to the background 
region in the original input image, and accordingly the image corresponding to the mixed region in the foreground 
component image becomes a dark image, for example, as compared with the image of the background region. 
[0919] The correction unit 2003 corrects pixel values of the pixels corresponding to the mixed region in the back- 
ground component image by multiplying each of pixel values of the pixels corresponding to the mixed region in the 
background component image by a constant corresponding to the mixture ratio a. 

[0920] For example, in the event that the background component image shown in Fig. 138 is input, the correction 
unit 2003 multiplies the pixel value C01 by 5/4, multiplies the pixel value C02 by 5/3, multiplies the pixel value C11 by 
5/3, and multiplies the pixel value C12 by 5/4. In order to match with the pixel position of the foreground component 
image which has been subjected to removal of movement blurring shown by way of an example shown in Fig. 135, 
the correction unit 2003 sets the pixel value C03 through pixel value C11 to 0. 

[0921] As described above, the correction unit 2003 corrects pixel values of the pixels corresponding to the mixed 
region in the background component image, and also adjusts the pixel position with regard to the foreground component 
image which has been subjected to removal of movement blurring. 

[0922] Fig. 140 is a block diagram which illustrates the configuration of the movement-blurring-removed-image 
processing unit 2004 for generating a coefficient set which is used in class classification adaptation processing for 
generating an even higher resolution image in the spatial direction. For example, the movement-blurring-removed- 
image processing unit 2004 of which the configuration is shown in Fig. 140 generates a coefficient set which is used 
in class classification adaptation processing for generating a HD image from a SD image based upon the input HD 
image. 

[0923] Background component tutor image frame memory 3001 stores the corrected background component image 
of the tutor image supplied from the correction unit 2003. The background component tutor image frame memory 3001 
supplies the stored background component image of the tutor image to a weighted averaging unit 3003-1 and a learning 
unit 3006-1. 

[0924] Foreground component tutor image frame memory 3002 stores the foreground component image of the tutor 
image, which has been subjected to removal of movement blurring, supplied from the movement blurring removal unit 
2002. The foreground component tutor image frame memory 3002 supplies the stored foreground component image 
of the tutor image to a weighted averaging unit 3003-2 and a learning unit 3006-2. 

[0925] The weighted averaging unit 3003-1 generates a SD image which is a student image by 1/4 weighted-aver- 
aging the background component image of a tutor image which is a HD image, and supplies the generated SD image 
to background component student image frame memory 3004. 

[0926] For example, the weighted averaging unit 3003-1 takes four pixels of 2 x 2 (width x height) (portions repre- 
sented by white circles in the drawing) as one increment in the tutor image as shown in Fig. 85, adds pixel values of 
four pixel of each increment, and the sum is divided by 4. The weighted averaging unit 3003-1 sets the 1/4 weighted 
averaged results described above for the pixel of the student image positioned at the center of each increment (which 
are the portions represented by solid circles in the drawing). 

[0927] The background component student image frame memory 3004 stores the student image corresponding to 
the background component image of the tutor image supplied from the weighted averaging unit 3003-1 . The background 
component student image frame memory 3004 supplies the student image corresponding to the background compo- 
nent image of the tutor image stored therein to the learning unit 3006-1 . 

[0928] The weighted averaging unit 3003-2 generates a SD image which is a student image by 1/4 weighted-aver- 
aging the foreground component image of a tutor image which is a HD image supplied from the foreground component 
tutor image frame memory 3002, for example, and supplies the generated SD image to foreground component student 
image frame memory 3005. 

[0929] The foreground component student image frame memory 3005 stores the student image which is a SD image, 
corresponding to the foreground component image of the tutor image supplied from the weighted averaging unit 3003-2. 
The foreground component student image frame memory 3005 supplies the student image corresponding to the fore- 
ground component image of the tutor image stored therein to the learning unit 3006-2. 

[0930] The learning unit 3006-1 generates coefficient sets corresponding to the background component image based 
upon the background component image of the tutor image supplied from the background component tutor image frame 
memory 3001 and the student image corresponding to the background component image of the tutor image supplied 
from the background component student image frame memory 3004, and supplies the generated coefficient sets to 
coefficient set memory 3007. 

[0931 ] The learning unit 3006-2 generates coefficient sets corresponding to the foreground component image based 
upon the foreground component image of the tutor image supplied from the foreground component tutor image frame 
memory 3002 and the student image corresponding to the foreground component image of the tutor image supplied 
from the foreground component student image frame memory 3005, and supplies the generated coefficient sets to 
coefficient set memory 3007. 
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[0932] The coefficient set memory 3007 stores the coefficient sets corresponding to the background component 
image supplied from the learning unit 3006-1 and the foreground component image supplied from the learning unit 
3006-2. 

[0933] In the event that there is no need to differentiate the learning unit 3006-1 and the learning unit 3006-2 indi- 

s vidually, these will be simply referred to as a learning unit 3006 below. 

[0934] Fig. 141 is a block diagram illustrating the configuration of the learning unit 3006. 
[0935] The class classification unit 3031 is configured of a class tap obtaining unit 3051 and a waveform classification 
unit 3052, and performs class classification of the pixels of interest, which are the pixels at interest, in the input student 
image. The class tap obtaining unit 3051 obtains a predetermined number of class taps which are pixels in the student 

10 image corresponding to the pixel of interest, and supply the obtained class taps to the waveform classification unit 3052 . 
[0936] The waveform classification unit 3052 executes class classification processing for classifying input signals 
into several classes based on the characteristics thereof, and classify the pixel of interest into one class of the 512 
classes, based on the class tap, and supplies class Nos. corresponding to the classes into which classification has 
been made, to a prediction tap obtaining unit 3032. 

'5 [0937] The prediction tap obtaining unit 3032 obtains from the pixels of the student image a prediction tap which is 
an increment for calculating predicted values for the original image (tutor image) corresponding to the class, based on 
the class No., and supplies the obtained prediction tap and class No. to a corresponding pixel obtaining unit 3033. 
[0938] The corresponding pixel obtaining unit 3033 obtains the pixel value of the pixel in the tutor image correspond- 
ing to the pixel value to be predicted, based on the prediction tap and the class No., and supplies the prediction tap, 

20 class No., and pixel value of the pixel in the tutor image corresponding to the pixel value to be predicted, to a normal 
equation generating unit 3034. 

[0939] The norma! equation generating unit 3034 generates a normal equation for calculating a coefficient set used 
in adaptation processing, corresponding to the relation between the prediction tap and the pixel value to be predicted, 
based on the prediction tap, class No., and pixel value of the pixel in the tutor image corresponding to the pixel value 
25 to be predicted, and supplies the generated norma! equation to a coefficient calculation unit 3035, along with the class 
No. 

[0940] The coefficient calculation unit 3035 solves the normal equation supplied from the normal equation generating 
unit 3034, and calculates a coefficient set to be used in the adaptation processing, corresponding to the class into 
which classification has been made. 
30 [0941] The coefficient calculation unit 3035 supplies the calculated coefficient set to coefficient set memory 7025, 
along with the class No. 

[0942] An arrangement may be made wherein the normal equation generating unit 3034 generates a matrix corre- 
sponding to such a normal equation, and the coefficient calculation unit 3035 calculates the coefficient set, based on 
the generated matrix. 

35 [0943] Fig. 1 42 is a diagram which describes a coefficient set generated by the movement-blurring-removed-image 
processing unit 2004 of which configuration is shown in Fig. 1 40. The region specifying unit 1 03 specifies the foreground 
region, the background region, the covered background region, and the uncovered background region in the input 
image. 

[0944] The input image, wherein the regions have been specified and the mixture ratio a has been detected by the 
<o mixture ratio calculation unit 1 04, is separated into the foreground component image and the background component 
image by the foreground/background separation unit 2001 . 

[0945] The movement blurring is removed from the separated foreground component image by the movement blur- 
ring removal unit 2002. The pixel values corresponding to the mixed region in the separated background component 
image are corrected by the correction unit 2003 corresponding to the removal of the movement blurring of the fore- 

45 ground component image. 

[0946] The movement-blurring-removed-image processing unit 2004 calculates a coefficient set corresponding to 
the foreground component image and a coefficient set corresponding to the background component image, respec- 
tively, based upon the foreground component image which has been subjected to removal of movement blurring and 
the background component image which has been subjected to correction. 

so [0947] That is to say, the learning unit 3006-1 calculates a coefficient set corresponding to the background component 
image based upon the separated and corrected background component image, and the learning unit 3006-2 calculates 
a coefficient set corresponding to the foreground component image based upon the foreground component image 
which has been subjected to separation and removal of movement blurring. 

[0948] The coefficient set corresponding to the background component image is used for predicting the pixel values 
55 pf the image corresponding to the background component image in the class classification adaptation processing for 
predicting the pixel values, which is to be applied to the separated and corrected background component image. 
[0949] The coefficient set corresponding to the foreground component image is used for predicting the pixel values 
of the image corresponding to the foreground component image in the class classification adaptation processing for 
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predicting the pixel values, which is to be applied to the foreground component image which has been subjected to 
separation and removal of movement blurring. 

[0950] The movement blurring is added to the predicted image corresponding to the foreground component image. 
The predicted image corresponding to the background component image is corrected corresponding to addition of the 
5 movement blurring to the foreground component image. 

[0951] The predicted image corresponding to the corrected background component image and the predicted image 
corresponding to the foreground component image which has been subjected to addition of the movement blurring, 
are synthesized into a single predicted image. 

[0952] Referring to the flowchart shown in Fig. 1 43, description will be made with regard to the processing of learning 

10 for generating a coefficient set which is used in prediction of the pixel values by the class classification adaptation 
processing in the movement-blurring-removed-image processing unit 2004 of which configuration is shown in Fig. 1 40. 
[0953] In Step S3001 , the weighted averaging unit 3003-1 and the weighted averaging unit 3003-2 generate a student 
image corresponding to the background component image and a student image corresponding to the foreground com- 
ponent image. That is to say, the weighted averaging unit 3003-1 generates a student image corresponding to the 

is background component image of the tutor image by 1/4 weighted-averaging of the background component image of 
the tutor image stored in the background component tutor image frame memory 3001 , for example. 
[0954] The weighted averaging unit 3003-2 generates a student image corresponding to the foreground component 
image of the tutor image by 1/4 weighted-averaging of the foreground component image of the tutor image stored in 
the foreground component tutor image frame memory 3002, for example. 

20 [0955] In Step S3002, the learning unit 3006-1 generates a coefficient set corresponding to the background compo- 
nent image based upon the background component image of the tutor image stored in the background component 
tutor image frame memory 3001 and the student image corresponding to the background component image of the 
tutor image stored in the background component student image frame memory 3004. Details of the processing for 
generating of a coefficient set in Step S3002 will be described later with reference to the flowchart shown in Fig. 1 44. 

25 [0956] In Step S3003, the learning unit 3006-2 generates a coefficient set corresponding to the foreground component 
image based upon the foreground component image of the tutor image stored in the foreground component tutor image 
frame memory 3002 and the student image corresponding to the foreground component image of the tutor image 
stored in the foreground component student image frame memory 3005. 

[0957] In Step S3004, the learning unit 3006-1 and the learning unit 3006-2 output a coefficient set corresponding 
30 to the background component image and a coefficient set corresponding to the foreground component image to the 
coefficient set memory 3007, respectively. The coefficient set memory 3007 stores the coefficient set corresponding 
to the background component image, or the coefficient set corresponding to the foreground component image, and 
then the processing ends. 

[0958] As described above, the movement-blurring-removed-image processing unit 2004 of which configuration is 
35 shown in Fig. 1 40 can generate a coefficient set corresponding to the background component image and a coefficient 
set corresponding to the foreground component image. 

[0959] Note that it is needless to say that the processing in Step S3002 and Step S3003 may be performed serially 
or in parallel. 

[0960] Next, referring to Fig. 144, the processing for generating a coefficient set corresponding to the background 
40 component image performed by the learning unit 3006-1 , corresponding to Step S3002, will now be described. 

[0961] In Step S3021 , the learning unit 3006-1 judges whether or not there are any unprocessed pixels in the student 
image corresponding to the background component image, and in the event that judgment is made that there are 
unprocessed pixels in the student image corresponding to the background component image, the flow proceeds to 
Step S3022, and the pixel of interest is obtained from the student image corresponding to the background component 
45 image in raster scan sequence. 

[0962] In Step S3023, the class tap obtaining unit 3051 of the class tap classification unit 3031 obtains a class tap 
corresponding to the pixel of interest from the student image stored in the background component student image frame 
memory 3004. In Step S3024, the waveform classification unit 3052 of the class classification unit 3031 applies the 
ADRC processing to the class tap, this reduces the number of bits of pixels making up the class tap, and the pixel of 
50 interest is classified. In Step S3025, the prediction tap obtaining unit 3032 obtains a prediction tap corresponding to 
the pixel of interest from the student image stored in the background component student image frame memory 3004 
based upon the classified class. 

[0963] In Step S3026, the corresponding pixel obtaining unit 3033 obtains pixels corresponding to the pixel value 
which is to be predicted from the background component image of the tutor image stored in the background component 
55 tutor image frame memory 3001 based upon the classified class. 

[0964] In Step S3027, the normal equation generating unit 3034 adds the pixel values of pixels corresponding to the 
prediction tap and the pixel value which is to be predicted to the matrix for each class based upon the classified class, 
the flow returns to Step S3021 , and the learning unit 3006-1 repeats judgment whether or not unprocessed pixels exist. 
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The matrix for each class to which the pixel values of pixels corresponding to the prediction tap and the pixel value 
which is to be predicted is added, corresponds to the normal equations for calculating a coefficient set for each class. 
[0965] In Step S3021 , in the event that judgment is made that there are no unprocessed pixels in the student image, 
the flow proceeds to Step S3028, and the normal equation generating unit 3034 supplies the matrix for each class for 
which the pixel values of the pixel corresponding to the prediction tap and the pixel value which is to be predicted is 
set, to the coefficient calculation unit 3035. The coefficient calculation unit 3035 calculates a coefficient set for each 
class corresponding to the background component image by solving the matrix for each class, wherein the pixel values 
of pixels corresponding to the prediction tap and the pixel value which is to be predicted are set. 
[0966] Note that the coefficient set is not restricted to predicting the pixel values by linear prediction, rather, an 
arrangement may be made wherein the coefficient calculation unit 3035 calculates a coefficient set for predicting the 
pixel values by non-linear prediction. 

[0967] In Step S3029, the coefficient calculation unit 3035 outputs the coefficient set for each class, corresponding 
to the background component image to the coefficient set memory 3007, and the processing ends. 
[0968] As described above, the learning unit 3006-1 can generate the coefficient set corresponding to the background 
component image. 

[0969] The processing for generating of the coefficient set corresponding to the foreground component image by the 
learning unit 3006-2 corresponding to Step S3003 is the same as the processing described with reference to the 
flowchart shown in Fig. 144 except for using the foreground component image stored in the foreground component 
tutor image frame memory 3002 and the student image corresponding to the foreground component image stored in 
the foreground component student image frame memory 105, and accordingly, description thereof will be omitted. 
[0970] As described above, the movement-blurring-removed-image processing unit 2004 of which the configuration 
is shown in Fig. 1 40 can generate a coefficient set corresponding to the background component image which has been 
subjected to correction and a coefficient set corresponding to the foreground component image which has been removal 
of movement blurring individually. 

[0971] Fig. 145 is a block diagram which illustrates the configuration of the movement-blurring-removed-image 
processing unit 2004 for generating an even higher resolution image in the. spatial direction by performing the class 
classification adaptation processing. For example, the movement blurring removal processing unit 2004 of which the 
configuration is shown in Fig. 145 generates an HD'image by performing the class classification adaptation processing 
based upon the input image which is a SD image. 

[0972] Background component image frame memory 31 01 stores the background component image which has been 
subjected to correction supplied from the correction unit 2003. The background component image frame memory 31 01 
supplies the stored background component image to a mapping unit 3103-1 . 

[0973] Foreground component image frame memory 31 02 stores the foreground component image made up of pixels 
belonging to the foreground region, supplied from the movement blurring removal unit 2002. The foreground component 
image frame memory 3102 supplies the stored foreground component image to a mapping unit 3103-2. 
[0974] The mapping unit 3103-1 generates a predicted image corresponding to the background component image 
stored in the background component image frame memory 3101 by the class classification adaptation processing 
based upon the coefficient set corresponding to the background component image stored in the coefficient set memory 
3104. The mapping unit 3103-1 supplies the generated predicted image to a correction unit 3105. 
[0975] The correction unit 31 05 sets the pixel value of the predetermined pixel in the predicted image corresponding 
to the mixed region in the background component image corresponding to the movement blurring, which the movement 
blurring addition unit 3106 adds, to 0; or divides the pixel value of the predetermined pixel in the predicted image by 
the predetermined value corresponding to the movement blurring which is added. The correction unit 31 05 supplies 
the predicted image which has been subjected to correction described above to a synthesizing unit 3107. 
[0976] The mapping unit 3103-2 generates a predicted image corresponding to the foreground component image 
stored in the foreground component image frame memory 31 02 by the class classification adaptation processing based 
upon the coefficient set corresponding to the foreground component image stored in the coefficient set memory 3104. 
The mapping unit 3103-2 supplies the generated predicted image to the movement blurring addition unit 3106. 
[0977] The movement blurring addition unit 3106 adds movement blurring to the predicted image by providing the 
desired movement blurring adjustment amount v' t e.g., the movement blurring adjustment amount v' of which value is 
the half value of the movement amount v of the input image or the movement blurring adjustment amount v 1 having no 
relationship with the movement amount v. The movement blurring addition unit 3106 calculates the foreground com- 
ponent Fi/v* by dividing the pixel value Fi in the predicted image in the foreground component image which has subjected 
to removal of movement blurring by the movement blurring adjustment amount v\ calculates the sum of the foreground 
components Fi/v's, and generates the pixel value which movement blurring is added to. 

[0978] For example, in the event that the predicted image shown in Fig. 146 is input, and the movement blurring 
adjustment amount V is 3, the pixel value C02 is (FOIJ/v', the pixel value C03 is (F01 + F02)/v\ the pixel value C04 is 
(F01 + F02 + F03)/V, and the pixel value C05 is (F02 + F03 + F04)/v' as shown in Fig. 147. 
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[0979] The movement blurring addition unit 31 06 supplies the predicted image of the foreground component image 
which has been subjected to addition of movement blurring, to the synthesizing unit 3107. 

[0980] The synthesizing unit 3107 synthesizes the predicted image corresponding to the background component 
image which has been subjected to correction supplied from the correction unit 3105, and the predicted image corre- 
5 sponding to the foreground component image which has been subjected to addition of movement blurring supplied 
from the movement blurring addition unit 3106, and supplies synthesized predicted image to the frame memory 3108. 
[0981 ] The frame memory 31 08 stores the predicted image supplied from the synthesizing unit 31 07, and also outputs 
the stored image as an output image. 

[0982] In the event that there is no need to differentiate the mapping unit 31 03-1 and the mapping unit 31 03-2 indi- 
10 vidually, these will be simply referred to as the mapping unit 31 03 below. 

[0983] Fig. 148 is a block diagram which illustrates the configuration of the mapping unit 3103. 
[0984] The mapping unit 3131 comprises a class classification unit 3141 for performing class classification process- 
ing, and a prediction tap obtaining unit 3142 and a prediction computation unit 3143 for performing the adaptation, 
processing. 

15 [0985] The class classification unit 3141 comprises a class tap obtaining unit 3151 and a waveform classification 
unit 3152, and performs class classification for pixel of interest in the input image of either background component 
image or foreground component image. 

[0986] The class tap obtaining unit 3151 obtains a predetermined number of class taps corresponding to pixel of 
interest in the input image, and supplies the obtained class taps to the waveform classification unit 31 52. For example, 
20 the class tap obtaining unit 31 51 obtains nine class taps, and supplies the obtained class taps to the waveform clas- 
sification unit 3152. 

[0987] The waveform classification unit 3152 reduces the number of bits of the pixels making up the class taps by 
applying the ADRC processing to the class taps, classifies the pixel of interest into one of the predetermined number 
of classes, and supplies the class No. corresponding to the classified class to the prediction tap obtaining unit 3142. 
25 For example, the waveform classification unit 3152 classifies the pixel of interest to one of 512 classes, and supplies 
the class No. corresponding to the classified class to the prediction tap obtaining unit 3142. 

[0988] The prediction tap obtaining unit 3142 obtains the predetermined number of prediction taps corresponding to 
the class from the input image based upon the class No., and supplies the obtained prediction taps and class No. to 
the prediction computation unit 3143. 

30 [0989] The prediction computation unit 31 43 obtains the coefficient set corresponding to the class, and corresponding 
to the input image, from the coefficient set corresponding to the background component image and coefficient set 
corresponding to the foreground component image, stored in the coefficient set memory 31 04 based upon the class 
No. The prediction computation unit 31 43 predicts a pixel value in the predicted image by linear prediction based upon 
the coefficient set and the prediction taps corresponding to the class, and corresponding to the input image. The pre- 

35 diction computation unit 3 143 supplies the predicted pixel value to the frame memory 31 32. 

[0990] Note that an arrangement may be made wherein the prediction computation unit 31 43 predicts the pixel value 
in the predicted image by non-linear prediction. 

[0991] The frame memory 3132 stores the predicted pixel values supplied from the mapping processing unit 3131 , 

and outputs the image made up of the predicted pixel values. 
40 [0992] Referring to the flowchart shown in Fig. 149, the processing for creation of the image by the movement- 

blurring-removed-image processing unit 2004 of which configuration is shown in Fig. 149 will be now described. 

[0993] In Step S3101, the mapping unit 3103-1 predicts the image corresponding to the background component 

image stored in the background component image frame memory 3101 by the class classification adaptation processing 

based upon the coefficient set corresponding to the background component image stored in the coefficient set memory 
45 3104. Details of the processing for prediction of the image corresponding to the background component image will be 

described later with reference to the flowchart shown in Fig. 150. 

[0994] In Step S31 02, the mapping unit 31 03-2 predicts the image corresponding to the foreground component image 
stored in the foreground component image frame memory 31 02 by the class classification adaptation processing based 
upon the coefficient set corresponding to the foreground component image stored in the coefficient set memory 31 04. 
so [0995] In Step S31 03, the correction unit 31 05 corrects the predicted image corresponding to the background com- 
ponent image. 

[0996] In Step S3104, the movement blurring addition unit 3106 adds movement blurring to the predicted image 
corresponding to the foreground component image. 

[0997] In Step S31 05, the synthesizing unit 3107 synthesizes the predicted image corresponding to the background 
55 component image with the predicted image corresponding to the foreground region. The synthesizing unit 31 07 supplies 
the synthesized image to the frame memory 3108. The frame memory 3108 stores the image supplied from the syn- 
thesizing unit 3107. 

[0998] In Step S31 06, the frame memory 31 08 outputs the stored and synthesized image, and the processing ends. 
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[0999] As described above, the image processing device having the movement-blurring-removed-image processing 
unit 2004 of which configuration is shown in Fig. 145 generates a predicted image corresponding to the background 
component image and a predicted image corresponding to the foreground component image which has been subjected 
to removal of movement blurring individually. 
5 [1000] Note that it is needless to say that the processing in Step S3101 and the processing in Step S3102 may be 
performed in a serial manner, as well as in a parallel manner. 

[1001] Referring to the flowchart shown in Fig. 150, the processing for prediction of the image corresponding to the 

background component Image by the mapping unit 3103-1 corresponding to Step S3 101 will be described. 

[1002] In Step S3121 , the mapping unit 31 03-1 judges whether or not there are any unprocessed pixels in the back- 

10 ground component image, and in the event that judgment is made that there are unprocessed pixels in the background 
component image, the flow proceeds to Step S3 122, and the mapping processing unit 3131 obtains the coefficient set 
corresponding to the background component image stored in the coefficient set memory 3104. In Step S3123, the 
mapping processing unit 3131 obtains a pixel of interest from the background component image stored in the back- 
ground component image frame memory 3101 in raster scan sequence. 

is [1003] In Step S3124, the class tap obtaining unit 3151 of the class classification unit 3141 obtains the class tap 
corresponding to the pixel of interest from the background component image stored in the background component 
image frame memory 3101. In Step S3 125, the waveform classification unit 3152 of the class classification unit 3141 
reduces the number of bits of pixels making up the class tap by applying the ADRC processing to the class tap, and 
performs class classification forthe pixel of interest. In Step S3126, the predication tap obtaining unit 3142 obtains the 

20 prediction tap corresponding to the pixel of interest from the background component image stored in the background 
component image frame memory 3101 based upon the classified class. 

[1004] In Step S3127, the prediction computation unit 3143 predicts pixel values of predicted image by linear pre- 
diction based upon the coefficient set and the prediction tap, corresponding to the background component image and 
the classified class. 

25 [1005] Note that the prediction computation unit 3143 may predict the pixel values of the predicted image by non- 
linear prediction, as well as to by linear prediction. 

[1006] In Step S3128, the prediction computation unit 3143 outputs the predicted pixel value to the frame memory 
31 32 . The frame memory 31 32 stores the pixel value supplied from the prediction computation unit 31 43. The procedure 
returns to Step S3121 , and judgment whether or not any unprocessed pixels exist is repeated. 
30 [1007] In Step S3121, in the event that judgment is made that there is no unprocessed pixel in the background 
component image, the flow proceeds to Step S3129, the frame memory 3132 outputs the stored predicted image 
corresponding to the background component image, and processing ends. 

[1008] As described above, the mapping unit 3103-1 can predict the image corresponding to the background com- 
ponent image based upon the corrected background component image. 

35 [1 009] The processing for generating of the predicted image corresponding to the foreground component image by 
the mapping unit 31 03-2 corresponding to Step S31 02 is the same as the processing described with reference to the 
flowchart shown in Fig. 150 except for using the foreground component image stored in the foreground component 
image frame memory 31 02 and the coefficient set corresponding to the foreground component image, and accordingly, 
description thereof will be omitted. 

*o [1010] As described above, the movement-blurring-removed-image processing unit 2004 of which configuration is 
shown in Fig. 1 45 can generate a predicted image corresponding to the background component image and a predicted 
image corresponding to the foreground component image which has been subjected to removal of movement blurring 
individually. 

[1011] Fig. 151 is a block diagram which illustrates the configuration of the movement-blurring-removed-image 
45 processing unit 2004 for applying edge enhancement processing with different effects for each background component 
image, or each foreground component image. 

[1012] Background component image frame memory 3201 stores the corrected background component image sup- 
plied from the correction unit 2003. The background component image frame memory 3201 supplies the stored back- 
ground component image to an edge enhancing unit 3203-1 . 
so [1013] Foreground component image frame memory 3202 stores the foreground component image which has been 
subjected to removal of movement blurring, supplied from the movement blurring removal unit 2002. The foreground 
component image frame memory 3202 supplies the stored foreground component image to an edge enhancing unit 
3203-2. 

[1014] The edge enhancing unit 3203-1 applies the processing of edge enhancement suitable for the background 
55 component image to the background component image stored in the background component image frame memory 
3201. 

[1015] For example, the edge enhancing unit 3203-1 performs the processing of edge enhancement which further 
enhances the edge for the background component image which is a still image as compared with the foreground 
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component image. Thus the sense-of-resolution of the background component image can be improved without unnat- 
ural degradation of the image occurring in the event of applying the processing of edge enhancement to images con- 
taining noise. 

[1016] The edge enhancing unit 3203-1 supplies the background component image which has been subjected to 
edge enhancement to a correction unit 3204. 

[1 01 7] The correction unit 3204 sets the pixel value of pixel in the mixed region in the background component image 
to 0, or divides the pixel value of the pixel in the mixed region by the predetermined value corresponding to the move- 
ment blurring which is to be added, corresponding to the movement blurring added by a movement blurring addition 
unit 3205. The correction unit 3204 supplies the image corrected as described above, to a synthesizing unit 3206. 
[1018] The edge enhancing unit 3203-2 applies the processing of edge enhancement suitable for the foreground 
component image, to the foreground component image stored in the foreground component image frame memory 3202. 
[1019] For example, the edge enhancing unit 3203-2 compares the foreground component image with the back- 
ground component image, and performs the processing of edge enhancement of which degree is less than that for the 
background component image. Thus the unnatural degradation in the image can be reduced as well as improving the 
sense-of-resolution in the foreground component image even if the foreground component image which has been 
subjected to removal of movement blurring contains noise. 

[1020] The edge enhancing unit 3203-2 supplies the foreground component image which has been subjected to 
edge enhancement to the movement blurring addition unit 3205. 

[1 021 ] The movement blurring addition unit 3205 adds movement blurring to the foreground component image which 
has been subjected to edge enhancement, and supplies the foreground component image which has been subjected 
to addition of movement blurring to a synthesizing unit 3206. 

[1022] The synthesizing unit 3206 synthesizes the background component image which has been subjected to edge 
enhancement and correction, supplied from the correction unit 3204, with the foreground component image which has 
been subjected to edge enhancement and addition of movement blurring, supplied from the movement blurring addition 
unit 3205, and supplies the synthesized predicted image to frame memory 3207. 

[1023] The frame memory 3207 stores the synthesized predicted image supplied from the synthesizing unit 3206, 
and also outputs the stored image as an output image. 

[1024] As described above, the movement-blurring-removed-image processing unit 2004 of which configuration is 
shown in Fig. 151 applies the edge enhancement processing corresponding to the nature of each image, for each 
background component image or each foreground component image, and accordingly the sense-of-resolution of the 
image is improved without degrading the image unnaturally. 

[1025] In the event that there is no need to differentiate the edge enhancing unit 3203-1 and the edge enhancing 
unit 3203-2 individually, these will be referred to as the edge enhancing unit 3203 below. 

[1026] For example, the edge enhancing unit 3203-1 has the same configuration as the edge enhancing unit 907, 
and applies edge enhancement processing with a greater degree of edge enhancement to the background component 
image. The edge enhancing unit 3203-2 has the same configuration as the edge enhancing unit 907, and applies edge 
enhancement processing with a relatively weaker degree of edge enhancement to the foreground component image. 
[1027] As described above, the edge enhancing unit 3203-1 and the edge enhancing unit 3203-2 applies the edge 
enhancement processing corresponding to the nature of the foreground component image or the background compo- 
nent image, to each foreground component image or each background component image, based upon the different 
filter coefficients or the gain adjustment coefficients, for example. 

[1028] Fig. 152 is a diagram which describes the processing in the movement-blurring-removed-image processing 
unit 2004 of which configuration is shown in Fig. 151 . 

[1 029] The specifying unit 1 03 specifies the foreground region, uncovered background region, covered background 
region, and background region in the input image. The input image of which regions are specified, is separated into 
the background component image and foreground component image by the foreground/background separation unit 
2001 . 

[1 030] The movement blurring removal unit 2002 removes movement blurring from the separated foreground com- 
ponent image. The correction unit 2003 corrects pixel values of the pixels corresponding to the mixed region in the 
separated background component image. 

[1031] The movement-blurring-removed-image processing unit 2004 of which configuration is shown in Fig. 151 
performs edge enhancement for each of the corrected background component image and the foreground component 
image which has been subjected to removal of movement blurring, corresponding to the nature of each image. 
[1032] The background component image which has been subjected to edge enhancement is corrected, correspond- 
ing to addition of the movement blurring to the foreground component image. The desired movement blurring is added 
to the foreground component image which has been subjected to edge enhancement. 

[1 033] The background component image which has been subjected to edge enhancement and correction, and the 
foreground component image which has been subjected to edge enhancement and addition of movement blurring, are 
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synthesized. 

[1 034] Referring to the flowchart shown in Fig. 1 53, the processing for edge enhancement by the movement-blurring- 
removed-image processing unit 2004 of which configuration is shown in Fig. 151 will be now described. 
[1035] In Step S3201 , the edge enhancing unit 3203-1 performs edge enhancement for the background component 
s image stored in the background component image frame memory 3201 by edge enhancement processing correspond- 
ing to the nature of the background component image. 

[1036] In Step S3202, the edge enhancing unit 3203-2 performs edge enhancement for the foreground component 
image stored in the foreground component image frame memory 3202 by the edge enhancement processing corre- 
sponding to the nature of the foreground component image. 
10 [1037] In Step S3203, the correction unit 3204 corrects pixel values of pixels in the background component image 
corresponding to addition of the movement blurring to the foreground component image. 

[1 038] In Step S3204, the movement blurring addition unit 3205 adds the desired movement blurring to the foreground 
component image. 

[1039] In Step S3205, the synthesizing unit 3206 synthesizes the background component image which has been 
'5 subjected to edge enhancement and correction, with the foreground component image which has been subjected to 

edge enhancement and addition of movement blurring. The synthesizing unit 3206 supplies the synthesized image to 

the frame memory 3207. The frame memory 3207 stores the image supplied from the synthesizing unit 3206. 

[1040] In Step S3206, the frame memory 3207 outputs the stored and synthesized image, and the processing ends. 

[1041] As described above, the movement-blurring-removed-image processing unit 2004 of which configuration is 
20 shown in Fig. 151 can perform the edge enhancement processing for each background component image and each 

foreground component image corresponding to the nature of each, and accordingly the sense-of-resolution can be 

improved without unnatural degradation in the image occurring. 

[1042] Note that it is needless to say that the processing in Step S3201 and Step S3202 may be performed serially 
or in parallel. 

25 [1043] Also, the processing which the movement-blurring-removed-image processing unit 2004 executes is not re- 
stricted to generating coefficients corresponding to SD images and HD images, or generating HD images from SD 
images, and may be arranged to generate coefficients for generating images with higher resolution in the spatial di- 
rection, and generate images with higher resolution in the spatial direction, for example. Further, the movement-biurring- 
removed-image processing unit 2004 may execute processing for generating images with higher resolution in the time 

30 direction. 

[1044] Note that the movement-blurring-removed-image processing unit 2004 is not restricted to class classification 
adaptation processing or edge enhancement processing, and may be arranged to execute other processing, such as, 
for example, conversion to an image size of a desired size, extracting color signals such as RGB, removing noise, 
compressing images, encoding, and so forth, for each image of specified regions. For example, the compression ratio 
35 can be increased with little deterioration of the image over conventional arrangements by the movement-blurring- 
removed-image processing unit 2004 compressing images of each of the regions with low compression ratio in direc- 
tions following movement vectors and high compression ratio in directions orthogonal to movement vectors, based on 
movement vectors corresponding to images of each of the regions. 

[1045] Also, an arrangement may be made wherein, in the event that the background object is moving, the image 
40 processing device removes the movement blurring contained in the background component image, so as to execute 

processing on a background component image from which movement blurring has been removed. 

[1046] Fig. 154 is a block diagram illustrating another configuration of the functions of the image processing device 

for separating an input image and processing each separated image. While the image processing device shown in Fig . 

1 1 9 performs region specification and calculation of the mixture ratio a serially, the image processing device shown in 
45 Fig. 154 performs region specification and calculation of the mixture ratio a in parallel. 

[1 047] Portions which are the same as the function in the block diagram shown in Fig. 1 1 9 are denoted with the same 

numerals, and description thereof will be omitted. 

[1048] The input image is supplied to the object extracting unit 101 , region specifying unit 103, mixture ratio calcu- 
lating unit 1101 , and foreground/background separation unit 3501 . 

50 [1049] Based on an input image, the mixture ratio calculating unit 1101 calculates an estimated mixture ratio in a 
case wherein a pixel is assumed to belong to the covered background region, and an estimated mixture ratio in a case 
wherein the pixel is assumed to belong to the uncovered background region, for each of the pixels contained in the 
input image, and supplies the estimated mixture ratio in a case wherein the pixel is assumed to belong to the covered 
background region and the estimated mixture ratio in a case wherein the pixel is assumed to belong to the uncovered 

55 background region, thus calculated, to the foreground/background separation unit 3501 . 

[1050] Fig. 155 is a block diagram which illustrates one example of the configuration of the foreground/background 
separation unit 3501. 

[1051] The same portions as the movement blurring removal unit 2002 shown in Fig. 124 are denoted by the same 
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reference numerals, and description thereof will be omitted. 

[1052] A selecting unit 3521 selects one or the other of the estimated mixture ratio in a case wherein the pixel is 
assumed to belong to the covered background region and the estimated mixture ratio in a case wherein the pixel is 
assumed to belong to the uncovered background region, supplied from the mixture ratio calculating unit 1101, based 
5 on the region information supplied from the region specifying unit 103, and supplies the selected estimated mixture 
ratio to the separating unit 2601 as mixture ratio a. 

[1 053) The separation unit 2601 extracts the foreground components and the background components from the pixel 
values of the pixels belonging to the mixed region based upon the mixture ratio a supplied from the selection unit 3521 
and the region information, and separates into the background components in the uncovered background region, the 
io foreground components in the uncovered background region, the background components in the covered background 
region, and the foreground components in the covered background region. 

[1054] The configuration of the separation unit 2601 may be the same as the configuration shown in Fig. 129. 
[1055] As described above, the image processing device of which configuration is shown in Fig. 154 can perform 
processing for each background component image and each foreground component image, corresponding to the nature 
15 of each image. 

[1056] As described above, with the image processing device according to the present invention, an input image is 
separated into a background component image and a foreground component image, and processing suitable for the 
separated images is executed, so images with higher resolution can be generated without generating unnatural images, 
for example. 

20 [1057] Note that while the movement of the object which is the foreground has been described as being from the left 
to the right, it is needless to say that this is not restricted to that direction. 

[1058] In the above, an example has been given of a case of projecting images in real space having three-dimensional 
space and time-axis information onto time-space having two-dimensional space and time-axis information, using a 
video camera, but the present invention is not restricted to this example, and may be applied to cases of projecting a 

25 greater amount of first information of a first dimension onto less second information of a second dimension. 

[1059] Note that the sensor is not restricted to a CCD, and may be a sensor which is a solid-state image-taking 
device, e.g., a CMOS (Complementary Metal Oxide Semiconductor (complementary metal oxide film semiconductor)), 
BBD (Bucket Brigade Device), CID (Charge Injection Device), or CPD (Charge Priming Device) or the like, and is not 
restricted to a sensor wherein detecting elements are arrayed in a matrix fashion, but may rather be a sensor wherein 

30 the detecting elements are arrayed in a row. 

[1060] The recording medium storing the program for executing the signal processing of the present invention is not 
only configured of packaged media such as a magnetic disk 91 (including floppy (Registered Trademark) disks), optical 
disk 92 (including CD-ROMs (Compact Disc-Read Only Memory), DVDs (Digital Versatile Disc)), magneto-optical disk 
93 (including MDs (Mini-Disc) (Registered Trademark)), or semiconductor memory 94 or the like, storing the program, 

35 to be distributed separately from the computer as shown in Fig. 1 0 for providing the program to users, but is configured 
of ROM 72 or a hard disk included in the storage unit 78 or the like storing the program, provided to the user in the 
state of being assembled into the computer beforehand. 

[1 061] Also, in the present Specification, the steps describing the program recorded in the recording medium includes 
processing which is executed in the time-sequence following the described order, of course, and also processing which 
40 is executed in parallel or individually, even if not processed in time-sequence. 

Industrial Applicability 

[1062] According to the present invention, images can be processed corresponding to the mixing of background 
45 images and images of moving objects. 



Claims 

50 1 . An image processing device for processing input image data made up of a predetermined number of pieces of 
pixel data obtained by an image-taking device having a predetermined number of pixels having time-integration 
effects, said image processing device comprising: 

region specifying means for specifying, based on said input image data, one or the other of a mixed region 
55 made up of a mixture of a foreground object component configuring foreground objects and a background 

object component configuring background objects, and a non-mixed region made up of one of a foreground 
region made up of said foreground object component and a background region made up of a background 
object component configuring said background objects, and outputting region specifying information corre- 
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sponding to the results of specifying; 

separating means for separating said input image data in at least said mixed region into said foreground object 
component and said background object component, corresponding to said region specifying information; and 
processing means for individually processing said foreground object component and said background object 
component, corresponding to the results of separation. 

An image processing device according to Claim 1 , further comprising removing means for removing movement 
blurring of at least one of said foreground object component and said background object component; 

wherein said processing means individually process said foreground object component and said background 
object component which have been subjected to movement blurring removal. 

An image processing device according to Claim 1 , wherein said region specifying means further specify a covered 
background region and an uncovered background region, and output said region specifying information corre- 
sponding to the results of specifying; 

and wherein said separating means separate said input image data into said foreground object component 
and said background object component in said covered background region and said uncovered background region. 

An image processing device according to Claim 1 , wherein said processing means generate coefficients used for 
class classification adaptation processing, for each of said foreground object component and said background 
object component. 

An image processing device according to Claim 1 , 

wherein said processing means generate output image data for each of said foreground object component and 
said background object component, by class classification adaptation processing. 

An image processing device according to Claim 1 , 

wherein said processing means perform edge enhancement for each of said foreground object component and 
said background object component. 

An image processing device according to Claim 1 , further comprising: 

foreground component image generating means for generating a foreground component image by synthesiz- 
ing said foreground object component separated in said mixed region and said pixel data of said foreground 
region; and 

background component image generating means for generating a background component image by synthe- 
sizing said background object component separated in said mixed region and said pixel data of said back- 
ground region; 

wherein said processing means individually process said foreground component image and said background 
component image which are generated. 

An image processing method for processing input image data made up of a predetermined number of pieces of 
pixel data obtained by an image-taking device having a predetermined number of pixels having time-integration 
effects, said method comprising: 

a region specifying step for specifying, based on said input image data, one or the other of a mixed region 
made up of a mixture of a foreground object component configuring foreground objects and a background 
object component configuring background objects, and a non-mixed region made up of one of a foreground 
region made up of said foreground object component and a background region made up of a background 
object component configuring said background objects, and outputting region specifying information corre- 
sponding to the results of specifying; 

a separating step for separating said input image data in at least said mixed region into said foreground object 
component and said background object component, corresponding to said region specifying information; and 
a processing step for individually processing said foreground object component and said background object 
component, corresponding to the results of separation. 

An image processing method according to Claim 8, further comprising a removing step for removing movement 
blurring of at least one of said foreground object component and said background object component; 
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wherein, in said processing step, said foreground object component and said background object component 
which have been subjected to movement blurring removal are individually processed. 

10. An image processing method according to Claim 8, wherein, in said region specifying step, a covered background 
region and an uncovered background region are further specified, and said region specifying information corre- 
sponding to the results of specifying is output; 

and wherein, in said separating step, said input image data is separated into said foreground object compo- 
nent and said background object component in said covered background region and said uncovered background 
region. 

11. An image processing method according to Claim 8, wherein, in said processing step, coefficients used for class 
classification adaptation processing are generated, for each of said foreground object component and said back- 
ground object component. 

12. An image processing method according to Claim 8, wherein, in said processing step, output image data is gener- 
ated for each of said foreground object component and said background object component, by class classification 
adaptation processing. 

13. An image processing method according to Claim 8, wherein, in said processing step, edge enhancement is per- 
formed for each of said foreground object component and said background object component. 

14. An image processing method according to Claim 8, further comprising: 

a foreground component image generating step for generating a foreground component image by synthesizing 
said foreground object component separated in said mixed region and said pixel data of said foreground region; 
and 

a background component image generating step for generating a background component image by synthe- 
sizing said background object component separated in said mixed region and said pixel data of said back- 
ground region; 

wherein, in said processing step, said foreground component image and said background component image 
which are generated, are individually processed. 

15. A recording medium storing a computer-readable program for processing input image data made up of a prede- 
termined number of pieces of pixel data obtained by an image-taking device having a predetermined number of 
pixels having time- integration effects, said program comprising: 

a region specifying step for specifying, based on said input image data, one or the other of a mixed region 
made up of a mixture of a foreground object component configuring foreground objects and a background 
object component configuring background objects, and a non-mixed region made up of one of a foreground 
region made up of said foreground object component and a background region made up of a background 
object component configuring said background objects, and outputting region specifying information corre- 
sponding to the results of specifying; 

a separating step for separating said input image data in at least said mixed region into said foreground object 
component and said background object component, corresponding to said region specifying information; and 
a processing step for individually processing said foreground object component and said background object 
component, corresponding to the results of separation. 

16. A recording medium according to Claim 15, said program further comprising a removing step for removing move- 
ment blurring of at least one of said foreground object component and said background object component; 

wherein, in said processing step, said foreground object component and said background object component 
which have been subjected to movement blurring removal are individually processed. 

17. A recording medium according to Claim 15, wherein, in said region specifying step, a covered background region 
and an uncovered background region are further specified, and said region specifying information corresponding 
to the results of specifying is output; 

and wherein, in said separating step, said input image data is separated into said foreground object compo- 
nent and said background object component in said covered background region and said uncovered background 
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region. 

18. A recording medium according to Claim 15, wherein, in said processing step, coefficients used for class classifi- 
cation adaptation processing are generated, for each of said foreground object component and said background 
object component 

19. A recording medium according to Claim 15, wherein, in said processing step, output image data is generated for 
each of said foreground object component and said background object component, by class classification adap- 
tation processing. 

20. A recording medium according to Claim 15, wherein, in said processing step, edge enhancement is performed for 
each of said foreground object component and said background object component. 

21. A recording medium according to Claim 15, said program further comprising: 

a foreground component image generating step for generating a foreground component image by synthesizing 
said foreground object component separated in said mixed region and said pixel data of said f oregrou nd region ; 
and 

a background component image generating step for generating a background component image by synthe- 
sizing said background object component separated in said mixed region and said pixel data of said back- 
ground region; 

wherein, in said processing step, said foreground component image and said background component image 
which are generated, are individually processed. 

22. A program for causing a computer, which processes input image data made up of a predetermined number of 
pieces of pixel data obtained by an image-taking device having a predetermined number of pixels having time- 
integration effects, to execute: 

a region specifying step for specifying, based on said input image data, one or the other of a mixed region 
made up of a mixture of a foreground object component configuring foreground objects and a background 
object component configuring background objects, and a non-mixed region made up of one of a foreground 
region made up of said foreground object component and a background region made up of a background 
object component configuring said background objects, and outputting region specifying information corre- 
sponding to the results of specifying; 

a separating step for separating said input image data in at least said mixed region into said foreground object 
component and said background object component, corresponding to said region specifying information; and 
a processing step for individually processing said foreground object component and said background object 
component, corresponding to the results of separation. 

23. A program according to Claim 22, further comprising a removing step for removing movement blurring of at least 
one of said foreground object component and said background object component; 

wherein, in said processing step, said foreground object component and said background object component 
which have been subjected to movement blurring removal are individually processed. 

24. A program according to Claim 22, wherein, in said region specifying step, a covered background region and an 
uncovered background region are further specified, and said region specifying information corresponding to the 
results of specifying is output; 

and wherein, in said separating step, said input image data is separated into said foreground object compo- 
nent and said background object component in said covered background region and said uncovered background 
region. 

25. A program according to Claim 22, wherein, in said processing step, coefficients used for class classification ad- 
aptation processing are generated, for each of said foreground object component and said background object 
component. 

26. A program according to Claim 22, wherein, in said processing step, output image data is generated for each of 
said foreground object component and said background object component, by class classification adaptation 
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processing. 

27. A program according to Claim 22, wherein, in said processing step, edge enhancement is performed for each of 
said foreground object component and said background object component. 

28. A program according to Claim 22, further comprising: 

a foreground component image generating step for generating a foreground component image by synthesizing 
said foreground object component separated in said mixed region and said pixel data of said foregrou nd region; 
and 

a background component image generating step for generating a background component image by synthe- 
sizing said background object component separated in said mixed region and said pixel data of said back- 
ground region; 

wherein, in said processing step, said foreground component image and said background component image 
which are generated, are individually processed. 

29. An image-taking device, comprising: 

image-taking means for outputting a subject image taken by an image-taking device having a predetermined 
number of pixels having time-integration effects as taken image data made up of a predetermined number of 
pieces of pixel data; 

region specifying means for specifying, based on said taken image data, one or the other of a mixed region 
made up of a mixture of a foreground object component configuring foreground objects and a background 
object component configuring background objects, and a non-mixed region made up of one of a foreground 
region made up of said foreground object component and a background region made up of a background 
object component configuring said background objects, and outputting region specifying information corre- 
sponding to the results of specifying; 

separating means for separating said taken image data in at least said mixed region into said foreground 
object component and said background object component, corresponding to said region specifying informa- 
tion; and 

processing means for individually processing said foreground object component and said background object 
component, corresponding to the results of separation. 

30. An image-taking device according to Claim 29, further comprising removing means for removing movement blurring 
of at least one of said foreground object component and said background object component; 

wherein said processing means individually process said foreground object component and said background 
object component which have been subjected to movement blurring removal. 

31 . An image-taking device according to Claim 29, wherein said region specifying means further specify a covered 
background region and an uncovered background region, and output said region specifying information corre- 

. sponding to the results of specifying; 

and wherein said separating means separate said taken image data into said foreground object component 
and said background object component in said covered background region and said uncovered background region . 

32. An image-taking device according to Claim 29, wherein said processing means generate coefficients used for 
class classification adaptation processing, for each of said foreground object component and said background 
object component. 

33. An image-taking device according to Claim 29, wherein said processing means generate output image data for 
each of said foreground object component and said background object component, by class classification adap- 
tation processing. 

34. An image-taking device according to Claim 29, wherein said processing means perform edge enhancement for 
each of said foreground object component and said background object component. 

35. An image-taking device according to Claim 29, further comprising: 
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foreground component image generating means for generating a foreground component image by synthesiz- 
ing said foreground object component separated in said mixed region and said pixel data of said foreground 
region; and 

background component image generating means for generating a background component image by synthe- 
sizing said background object component separated in said mixed region and said pixel data of said back- 
ground region; 

wherein said processing means individually process said foreground component image and said background 
component image which are generated. 
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FIG. 38A 



FIG. 38B 



FIG. 38C 



FIG. 38D 
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FIG. 39 
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FIG. 40 
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Description 

Technical Field 

[0001] The present invention relates to an image processing device and method, and an image-taking device, and 
particularly relates to an image processing device and method, and an image-taking device, which take into consider- 
ation difference between signals detected by sensors and the real world. 

Background Art 

[0002] One type of processing for generating images with higher resolution based on input images, is class classi- 
fication adaptation processing. An example of class classification adaptation processing is processing wherein coeffi- 
cients used in processing for generating images with higher resolution are generated beforehand, in the spatial direc- 
tion, and images are generated with higher resolution in the spatial direction based on the generated coefficients. 
[0003] Fig. 1 is a block diagram illustrating the configuration of a conventional image processing device for generating 
coefficients used in class classification adaptation processing for generating HD (High Definition) images from SD 
(Standard Definition) images. 

[0004] Frame memory 11 stores input images, which are HD images, in increments of frames. The frame memory 
11 supplies the stored HD images to a weighted averaging unit 12 and a corresponding pixel obtaining unit 16. 
[0005] The weighted averaging unit 12 performs 1/4 weighted averaging on the HD images stored in the frame 
memory 11 , generates SD images, and supplies the generated SD images to the frame memory 13. 
[0006] The frame memory 13 stores the SD images supplied from the weighted averaging unit 12 in increments of 
frames, and supplies the stored SD images to a class classification unit 14 and prediction tap obtaining unit 15. 
[0007] The class classification unit 14 is configured of a class tap obtaining unit 21 and a waveform classification 
unit 22, and performs class classification of pixels of interest which are the pixel of interest in the SD images stored in 
the frame memory 13. The class tap obtaining unit 21 obtains a predetermined number of class taps which are pixels 
of the SD image corresponding to the pixel of interest from the frame memory 1 3, and supplies the obtained class taps 
to the waveform classification unit 22. 

[0008] Fig. 2 is a diagram explaining the class taps obtained by the class tap obtaining unit 21 . As shown in Fig. 2, 
the class tap obtaining unit 21 obtains eleven class taps at predetermined positions. 

[0009] The waveform classification unit 22 classifies the pixel of interest into one class out of multiple classes, based 
on the class taps, and supplies a class No. corresponding to the classified class, to the prediction tap obtaining unit 
1 5. The waveform ciassif ication unit 22 classifies the pixel of interest Into one class out of 2048 classes, based on the 
eleven class taps. 

[0010] The prediction tap obtaining unit 15 obtains a predetermined number of prediction taps which are pixels of 
the SD image, corresponding to the classified class from the frame memory 13, based on the class No., and supplies 
the obtained prediction taps and class Nos. to a corresponding pixel obtaining unit 16. 

[001 1 ] Fig. 3 is a diagram explaining prediction taps which the prediction tap obtaining unit 1 5 obtains. As shown in 
Fig. 3, the prediction tap obtaining unit 15 obtains nine prediction taps at predetermined locations. 
[0012] The corresponding pixel obtaining unit 1 6 obtains, from the frame memory 1 1 , pixels of the HD image corre- 
sponding to the pixel values to be predicted, based on the prediction taps and the class Nos„ and supplies the prediction 
taps, class Nos., and the pixels of the HD image corresponding to the obtained pixel values to be predicted, to a normal 
equation generating unit 1 7. 

[0013] The normai equation generating unit 1 7 generates normal equations corresponding to relationships between 
prediction taps and pixel values to be predicted, corresponding to the classes, based on the prediction taps, class 
Nos., and the obtained pixel values to be predicted, and supplies the generated normal equations corresponding to 
the classes, to a coefficient calculating unit 18. 

[0014] The coefficient calculating unit 18 solves the normal equation supplied from the normal equation generating 
unit 1 7, calculates coefficient sets corresponding to each class, and supplies the calculated coefficient sets to coefficient 
set memory 19, along with the class Nos. 

[001 5] The coefficient set memory 1 9 stores the calculated coefficient sets corresponding to the classes, based on 
the class Nos. 

[001 6] Fig. 4 is a diagram explaining an overview of class classification adaptation processing. In class classification 
adaptation processing, a tutor image which is an HD image is used to generate a corresponding SD image, by 1/4 
weighted average processing. The generated SD image is called a student image. 

[0017] Next, a coefficient set for generating an HD image from the SD image is generated, based on the tutor image 
which is the HD image and the student image which is the corresponding SD image. The coefficient set is configured 
of coefficients for generating an HD image from an SD image, by linear prediction and the like. 
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[001 8] A quadruple-density image is generated from the coefficients set thus generated and the SD image, by linear 
prediction and the like. The processing for generating an image or the like with higher density, from a coefficient set 
and an input image, is also called mapping. 

[0019] SNR comparison, or visual qualitative evaluation is performed, based on the generated quadruple-density 
5 image and a corresponding HD image. 

[0020] A coefficient set generated from a particular tutor image and corresponding student image is called a self 

coefficient set of the particular tutor image and corresponding student image. Mapping using the self coefficient set is 

called self mapping. A coefficient set generated from multiple other tutor images and corresponding student images 

is called a cross coefficient set. 
10 [0021] On the other hand, with images obtained by a video camera taking a foreground subject which moves across 

a predetermined stationary background, movement blurring occurs in the event that the speed of movement of the 

object is relatively fast, and mixing of the foreground and background occurs. 

[0022] With conventional class classification adaptation processing, one set of coefficients is generated for all of the 
foreground, background, and portions where mixing between the foreground and background occurs, by learning 

15 processing such as described above, and mapping processing is executed based on the coefficient set. 

[0023] The conventional learning processing for generating coefficients used in the processing for generating HD 
images from SD images will be described, with reference to the flowchart shown in Fig. 6. In Step S11, an image 
processing device judges whether or not there are unprocessed pixels in the student image, and in the event that 
judgment is made that there are unprocessed pixels in the student image, the flow proceeds to Step S12, and pixels 

20 of interest are obtained from the student image, in order of raster scan. 

[0024] In Step S1 3, the class tap obtaining unit 21 of the class classification unit 1 4 obtains a class tap corresponding 
to the pixel of interest, from the student image stored in the frame memory 1 3. In Step S1 4, the waveform classification 
unit 22 of the class classification unit 14 performs class classification of the pixel of interest, based on the class tap. 
In Step S15, the prediction tap obtaining unit 15 obtains a prediction tap corresponding to the pixel of interest from the 

25 student image stored in the frame memory 13, based on the class into which classification has been made. 

[0025] In Step S16, the corresponding pixel obtaining unit 16 obtains a pixel corresponding to a pixel value to be 
predicted, from tutor data stored in the frame memory 11 , based on the class into which classification has been made. 
[0026] In Step S1 7, the normal equation generating unit 1 7 adds a pixel value of a pixel corresponding to the prediction 
tap and pixel value to be predicted to the matrix for each class, based on the class into which classification has been 

so made, the flow returns to Step S11 , and the image processing device repeats judgment regarding whether or not there 
are unprocessed pixels. The matrixes for each class to which the pixel value of a pixel corresponding to the prediction 
tap and pixel value to be predicted are added, correspond to the normal equation for calculating coefficients for each 
class. 

[0027] In the event that judgment is made in Step S11 that there are no unprocessed pixels in the student image, 
35 the flow proceeds to Step S18, wherein the normal equation generating unit 17 supplies the matrix for each class 
wherein a pixel value of a pixel corresponding to the prediction tap and pixel value to be predicted has been set, to the 
coefficient calculating unit 1 8. The coefficient calculating unit 1 B solves the matrix for each class wherein a pixel value 
of a pixel corresponding to the prediction tap and pixel value to be predicted has been set, and calculates a coefficient 
set for each class. 

40 [0028] In Step S1 9, the coefficient calculating unit 1 8 outputs the coefficient for each class that has been calculated, 
to the coefficient set memory 1 9. The coefficient set memory 1 9 stores a coefficient set for each class, and the process- 
ing ends. 

[0029] Fig. 7 is a block diagram illustrating the configuration of a conventional image processing device for generating 
HD images from SD images, by class classification adaptation processing. 
45 [0030] Frame memory 31 stores input images which are SD images, in increments of frames. The frame memory 
31 supplies the stored SD images to a mapping unit 32. 

[0031] The SD images input to the mapping unit 32 are supplied to a class classification unit 41 and a prediction tap 
obtaining unit 42. 

[0032] The class classification unit 41 is configured of a class tap obtaining unit 51 and a waveform classification 
so unit 52, and performs class classification of pixels of interest which are the pixel of interest in the SD images stored in 
the frame memory 31 . The class tap obtaining unit 51 obtains from the frame memory 31 a predetermined number of 
class taps corresponding to the pixel of interest, and supplies the obtained class taps to the waveform classification 
unit 52. 

[0033] The waveform classification unit 52 classifies the pixel of interest into one class out of multiple classes, based 
55 on the class taps, and supplies a class No. corresponding to the classified class, to the prediction tap obtaining unit 42. 
[0034] The prediction tap obtaining unit 42 obtains from the input image stored in the frame memory 31 a predeter- 
mined number of prediction taps corresponding to the classified class, based on the class No . , and supplies the obtained 
prediction taps and class Nos. to a prediction computation unit 43. 
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[0035] The prediction computation unit 43 obtains coefficient sets corresponding to classes from the coefficient sets 
stored in coefficient set memory 33, based on the class No. The prediction computation unit 43 predicts pixel values 
of predicted images by linear prediction, based on coefficient sets corresponding to classes, and prediction taps. The 
prediction computation unit 43 supplies the predicted pixel values to frame memory 34. 

[0036] The frame memory 34 stores predicted pixel values supplied from the prediction computation unit 43, and 
outputs an HD image wherein the predicted pixel values have been set. 

[0037] Fig. 8 is a diagram illustrating the pixel values of the input image, and the pixel values of the output image 
generated by class classification adaptation processing. In Fig. 8, the white squares indicate input signals, and the 
solid circles indicate output signals. As shown in Fig. 8, the image generated by the class classification adaptation 
processing contains waveforms lost in the bandwidth restriction of the SD image. In this sense, it can be said that 
processing for generating an image with higher resolution by the class classification adaptation processing creates 
resolution. 

[0038] The conventional processing for creating images, for generating HD images from SD image with an image 
processing device which executes processing for creating resolution by class classification adaptation processing, will 
be described with reference to the flowchart in Fig. 9. 

[0039] In Step S31, the image processing device judges whether or not there are unprocessed pixels in the input 
image, and in the event that judgment is made that there are unprocessed pixels in the input image, the flow proceeds 
to Step S32, where the mapping unit 32 obtains a coefficient set stored in the coefficient set memory 33. In Step S33, 
the image processing device obtains pixels of interest from the input image in raster scan order. 
[0040] In Step S34, the class tap obtaining unit 51 of the class classification unit 41 obtains a class tap corresponding 
to the pixel of interest, from the input image stored in the frame memory 31 . In Step S35, the waveform classification 
unit 52 of the class classification unit 41 performs class classification of the pixel of interest into one class, based on 
the class tap. 

[0041] In Step S36, the prediction tap obtaining unit 42 obtains a prediction tap corresponding to the pixel of interest 
from the input image stored in the frame memory 31 , based on the class into which classification has been made. 
[0042] In Step S37, the prediction computation unit 43 obtains a pixel value of a predicted image by linear prediction, 
based on the coefficient set corresponding to the class into which classification has been made, and the prediction tap. 
[0043] In Step S38, the prediction computation unit 43 outputs the predicted pixel value to the frame memory 34. 
The frame memory 34 stores the pixel value supplied from the prediction computation unit 43. The procedures return 
to Step S31 , and repeats judgement regarding whether or not there are unprocessed pixels. 
[0044] In the event that judgment is made in Step S31 that there are no unprocessed pixels in an input image, the 
flow proceeds to Step S39, where the frame memory 34 outputs the stored predicted image wherein predicted values 
are set, and the processing ends. 

[0045] Also, edge enhancing processing is used for converting the input image into an image with the sense-of- 
resolution enhanced even further. As with the class classification adaptation processing described above, the same 
processing is executed for the entire screen with the edge enhancement processing as well. 
[0046] However, in the event that objects move in front of still backgrounds, movement blurring occurs not only due 
to mixture of the moving object images itself, but also due to mixture of the moving object images and the background 
images. Conventionally, processing images corresponding to the mixing of the background image and the image of 
the moving object had not been given thought. 

Disclosure of Invention 

[0047] The present invention has been made in light of the above, and it is an object thereof to enable processing 
of images corresponding to the mixing of background images and images of the moving objects. 
[0048] An image processing device according to the present invention comprises: region specifying means for spec- 
ifying, based on the input image data, one or the other of a mixed region made up of a mixture of a foreground object 
component configuring foreground objects and a background object component configuring background objects, and 
a non-mixed region made up of one of a foreground region made up of the foreground object component and a back- 
ground region made up of a background object component configuring the background objects, and outputting region 
specifying information corresponding to the results of specifying; separating means for separating the input image data 
in at least the mixed region into the foreground object component and the background object component, corresponding 
to the region specifying information ; and processing means for individually processing the foreground object component 
and the background object component, corresponding to the results of separation. 

[0049] The image processing device may further comprise removing means for removing movement blurring of at 
least one of the foreground object component and the background object component, with the processing means 
individually processing the foreground object component and the background object component which have been 
subjected to movement blurring removal. 
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[0050] The region specifying means may further specify a covered background region and an uncovered background 
region, and output the region specifying information corresponding to the results of specifying, with the separating 
means separating the input image data into the foreground object component and the background object component 
in the covered background region and the uncovered background region. 

[0051] The processing means may generate coefficients used for class classification adaptation processing, for each 
of the foreground object component and the background object component. 

[0052] The processing means may generate output image data for each of the foreground object component and 
the background object component, by class classification adaptation processing. 

[0053] The processing means may perform edge enhancement for each of the foreground object component and 
the background object component. 

[0054] The image processing device may further comprise: 

foreground component image generating means for generating a foreground component image by synthesizing the 
foreground object component separated in the mixed region and the pixel data of the foreground region; and background 
component image generating means for generating a background component image by synthesizing the background 
object component separated in the mixed region and the pixel data of the background region; with the processing 
means individually processing the foreground component image and the background component image which are 
generated. 

[0055] An image processing method according to the present invention comprises: a region specifying step forspec- 
ifying, based on the input image data, one or the other of a mixed region made up of a mixture of a foreground object 
component configuring foreground objects and a background object, component configuring background objects, and 
a non-mixed region made up of one of a foreground region made up of the foreground object component and a back- 
ground region made up of a background object component configuring the background objects, and outputting region 
specifying information corresponding to the results of specifying; a separating step for separating the input image data 
in at least the mixed region into the foreground object component and the background object component, corresponding 
to the region specifying information; and a processing step for individually processing the foreground object component 
and the background object component, corresponding to the results of separation. 

[0056] The image processing method may further comprise a removing step for removing movement blurring of at 
least one of the foreground object component and the background object component, with the foreground object com- 
ponent and the background object component which have been subjected to movement blurring removal being indi- 
vidually processed in the processing step. 

[0057] In the region specifying step, a covered background region and an uncovered background region may be 
further specified, and the region specifying information corresponding to the results of specifying output, with the input 
image data being separated into the foreground object component and the background object component in the covered 
background region and the uncovered background region in the separating step. 

[0058] In the processing step, coefficients used for class classification adaptation processing may be generated, for 
each of the foreground object component and the background object component. 

[0059] In the processing step, output image data may be generated for each of the foreground object component 
and the background object component, by class classification adaptation processing. 

[0060] In the processing step, edge enhancement may be performed for each of the foreground object component 
and the background object component. 

[0061] The image processing method may further comprise: a foreground component image generating step for 
generating a foreground component image by synthesizing the foreground object component separated in the mixed 
region and the pixel data of the foreground region; and a background component image generating step for generating 
a background component image by synthesizing the background object component separated in the mixed region and 
the pixel data of the background region; with the foreground component image and the background component image 
which are generated, being individually processed in the processing step. 

[0062] A program in a recording medium according to the present invention comprises: a region specifying step for 
specifying, based on the input image data, one or the other of a mixed region made up of a mixture of a foreground 
object component configuring foreground objects and a background object component configuring background objects, 
and a non-mixed region made up of one of a foreground region made up of the foreground object component and a 
background region made up of a background object component configuring the background objects, and outputting 
region specifying information corresponding to the results of specifying; a separating step for separating the input 
image data in at least the mixed region into the foreground object component and the background object component, 
corresponding to the region specifying information; and a processing step for individually processing the foreground 
object component and the background object component, corresponding to the results of separation. 
[0063] The program in the recording medium may further comprise a removing step for removing movement blurring 
of at least one of the foreground object component and the background object component, with the foreground object 
component and the background object component which have been subjected to movement blurring removal being 
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individually processed in the processing step. 

[0064] In the region specifying step, a covered background region and an uncovered background region may be 
further specified, and the region specifying information corresponding to the results of specifying output, with the input 
image data being separated into the foreground object component and the background object component in the covered 
background region and the uncovered background region in the separating step. 

[0065] In the processing step, coefficients used for class classification adaptation processing may be generated, for 
each of the foreground object component and the background object component. 

[0066] In the processing step, output image data may be generated for each of the foreground object component 
and the background object component, by class classification adaptation processing. 

[0067] In the processing step, edge enhancement may be performed for each of the foreground object component 
and the background object component. 

[0068] The program in the recording medium may further comprise: a foreground component image generating step 
for generating a foreground component image by synthesizing the foreground object component separated in the mixed 
region and the pixel data of the foreground region; and a background component image generating step for generating 
a background component image by synthesizing the background object component separated in the mixed region and 
the pixel data of the background region; with the foreground component image and the background component image 
which are generated, being individually processed in the processing step. 

[0069] A program according to the present invention causes a computer to execute: a region specifying step for 
specifying, based on the input image data, one or the other of a mixed region made up of a mixture of a foreground 
object component configuring foreground objects and a background object component configuring background objects, 
and a non-mixed region made up of one of a foreground region made up of the foreground object component and a 
background region made up of a background object component configuring the background objects, and outputting 
region specifying information corresponding to the results of specifying; a separating step for separating the input 
image data in at least the mixed region into the foreground object component and the background object component, 
corresponding to the region specifying information; and a processing step for individually processing the foreground 
object component and the background object component, corresponding to the results of separation. 
[0070] The program may further comprise a removing step for removing movement blurring of at least one of the 
foreground object component and the background object component, with the foreground object component and the 
background object component which have been subjected to movement blurring removal being individually processed 
in the processing step. 

[0071] In the region specifying step, a covered background region and an uncovered background region may be 
further specified, and the region specifying information corresponding to the results of specifying output, with the input 
image data being separated into the foreground object, component and the background object component in the cov- 
ered background region and the uncovered background region in the separating step. 

[0072] In the processing step, coefficients used for class classification adaptation processing may be generated, for 
each of the foreground object component and the background object component. 

[0073] In the processing step, output image data may be generated for each of the foreground object component 
and the background object component, by class classification adaptation processing. 

[0074] In the processing step, edge enhancement may be performed for each of the foreground object component 
and the background object component. 

[0075] The program may further comprise: a foreground component image generating step for generating a fore- 
ground component image by synthesizing the foreground object component separated in the mixed region and the 
pixel data of the foreground region; and a background component image generating step for generating a background 
component image by synthesizing the background object component separated in the mixed region and the pixel data 
of the background region; with the foreground component image and the background component image which are 
generated, being individually processed In the processing step. 

[0076] An image-taking device according to the present invention comprises: image-taking means for outputting a 
subject image taken by an image-taking device having a predetermined number of pixels having time-integration effects 
as taken image data made up of a predetermined number of pieces of pixel data; region specifying means for specifying, 
based on the taken image data, one or the other of a mixed region made up of a mixture of a foreground object 
component configuring foreground objects and a background object component configuring background objects, and 
a non-mixed region made up of one of a foreground region made up of the foreground object component and a back- 
ground region made up of a background object component configuring the background objects, and outputting region 
specifying information corresponding to the results of specifying; separating means for separating the taken image 
data in at least the mixed region into the foreground object component and the background object component, corre- 
sponding to the region specifying information; and processing means for individually processing the foreground object 
component and the background object component, corresponding to the results of separation. 
[0077] The image-taking device may further comprise removing means for removing movement blurring of at least 
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one of the foreground object component and the background object component, with the processing means individually 
processing the foreground object component and the background object component which have been subjected to 
movement blurring removal. 

[0076] The region specifying means may further specify a covered background region and an uncovered background 
region, and output the region specifying information corresponding to the results of specifying, with the separating 
means separating the taken image data into the foreground object component and the background object component 
in the covered background region and the uncovered background region. 

[0079] The processing means may generate coefficients used for class classification adaptation processing, for each 
of the foreground object component and the background object component. 

[0080] The processing means may generate output image data for each of the foreground object component and 
the background object component, by class classification adaptation processing. 

[0081] The processing means may perform edge enhancement for each of the foreground object component and 

the background object component. 

[0082] The image-taking device may further comprise: 

foreground component image generating means for generating a foreground component image by synthesizing the 
foreground object component separated in the mixed region and the pixel data of the foreground region; and background 
component image generating means for generating a background component image by synthesizing the background 
object component separated in the mixed region and the pixel data of the background region, with the processing 
means individually processing the foreground component image and the background component image which are 
generated. 

[0083] One or the other of a mixed region made up of a mixture of a foreground object component configuring fore- 
ground objects and a background object component configuring background objects, and a non-mixed region made 
up of one of a foreground region made up of the foreground object component and a background region made up of 
a background object component configuring the background objects, are specified, region specifying information cor- 
responding to the results of specifying is output, the input image data in at least the mixed region is separated into the 
foreground object component and the background object component, corresponding to the region specifying informa- 
tion, and the foreground object component and the background object component are individually processed, corre- 
sponding to the results of separation. 

[0084] Thus, in the event that a moving object is photographed, for example, images can be processed corresponding 
to the mixing of background images and moving object images. 

Brief Description of the Drawings 

[0085] 

Fig. 1 is a block diagram illustrating the configuration of a conventional image processing device. 
Fig. 2 is a diagram explaining class taps. 
Fig. 3 is a diagram explaining prediction taps. 

Fig. 4 is a diagram describing the overview of class classification adaptation processing. 
Fig. 5 is a diagram explaining conventional coefficient sets. 
Fig. 6 is a flowchart explaining conventional learning processing. 

Fig. 7 is a block diagram illustrating the configuration of a conventional image processing device. 

Fig. 6 is a diagram illustrating pixel values of an output image generated by pixel values of an input image, and 

class classification adaptation processing. 

Fig. 9 is a flowchart explaining conventional processing for creating images. 

Fig. 1 0 is a block diagram illustrating the configuration of an embodiment of an image processing device according 
to the present invention. 

Fig. 11 is a block diagram illustrating a configuration of an image processing device. 

Fig. 12 is a diagram describing image-taking by a sensor. 

Fig. 13 is a diagram describing an arrangement of pixels. 

Fig. 14 is a diagram describing operation of a detecting device. 

Fig. 15A is a diagram describing an image obtained by taking an image of an object corresponding to a moving 
foreground, and an object corresponding to a still background. 

Fig. 15B is a diagram describing a model corresponding to an image obtained by taking an image of an object 
corresponding to a moving foreground, and an object corresponding to a still background. 
Fig. 16 is a diagram describing background region, foreground region, mixed region, covered background region, 
and uncovered background region. 

Fig. 17 is a model diagram which develops, over the time direction, the pixel values of pixels adjacently arrayed 
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in one row, in an image wherein an object corresponding to a still foreground and an object corresponding to a still 
background are subjected to image-taking. 

Fig. 1 8 is a model diagram wherein the pixel values are developed over the time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 1 9 is a model diagram wherein the pixel values are developed over the time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 20 is a model diagram wherein the pixel values are developed over the time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 21 is a diagram illustrating an example of extracting pixels of the foreground region, background region, and 
mixed region. 

Fig. 22 is a diagram illustrating how pixels correspond to a model wherein pixel values are developed over the 
time direction. 

Fig. 23 is a model diagram wherein the pixel values are developed over the time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 24 is a model diagram wherein the pixel values are developed over the time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 25 is a model diagram wherein the pixel values are developed over the time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 26 is a model diagram wherein the pixel values are developed overthe time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 27 is a model diagram wherein the pixel values are developed overthe time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 28 is a diagram illustrating the correlation between a divided image, and a model diagram wherein the pixel 

values of pixels are developed over the time direction. 

Fig. 29 is a diagram illustrating an example of a divided image. 

Fig. 30 is a diagram illustrating an example of a divided image. 

Fig. 31 is a flowchart explaining the processing of images with the image processing device according to the 
present invention. 

Fig. 32 is a block diagram illustrating an example of the configuration of the region specifying unit 1 03. 
Fig. 33 is a diagram describing an image wherein an object corresponding to the foreground is moving. 
Fig. 34 is a model diagram wherein the pixel values are developed overthe time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 35 is a model diagram wherein the pixel values are developed overthe time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 36 is a model diagram wherein the pixel values are developed overthe time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 37 is a diagram describing conditions for region judgment. 

Fig. 38A is a diagram illustrating an example of the results of region specification made by the region specifying 
unit 103. 

Fig. 38B is a diagram illustrating an example of the results of region specification made by the region specifying 
unit 103. 

Fig. 38C is a diagram illustrating an example of the results of region specification made by the region specifying 
unit 103. 

Fig. 38D is a diagram illustrating an example of the results of region specification made by the region specifying 
unit 103. 

Fig. 39 is a diagram illustrating an example of the results of region specification made by the region specifying unit 
103. 

Fig. 40 is a flowchart explaining processing for region specifying. 

Fig. 41 is a block diagram illustrating another example of the configuration of the region specifying unit 103. 
Fig. 42 is a model diagram wherein the pixel values are developed overthe time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 43 is a diagram illustrating an example of a background region image. 

Fig. 44 is a block diagram illustrating the configuration of a binary object image extracting unit 302. 

Fig. 45A is a diagram describing calculating of correlation values. 

Fig. 45B is a diagram describing calculating of correlation values. 

Fig. 46A is a diagram describing calculating of correlation values. 

Fig. 46B is a diagram describing calculating of correlation values. 

Fig. 47 is a diagram illustrating an example of a binary object image. 
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Fig. 48 is a block diagram illustrating the configuration of a time change detecting unit 303. 
Fig. 49 is a diagram describing judgment of a region judgment unit 342. 

Fig. 50 is a diagram illustrating an example of judgment made by the time change detecting unit 303. 

Fig. 51 is a flowchart describing processing for region specification by the region Judgment unit 103. 

Fig. 52 is a flowchart for describing the processing for region specification in detail. 

Fig. 53 is a block diagram illustrating yet another configuration of the region specifying unit 103. 

Fig. 54 is a block diagram describing the configuration of a robustification unit 361. 

Fig. 55 is a diagram describing movement compensation of a movement compensation unit 381 . 

Fig. 56 is a diagram describing movement compensation of a movement compensation unit 381 . 

Fig. 57 is a flowchart describing the processing for region specification. 

Fig. 58 is a flowchart describing details of processing for robustification. 

Fig. 59 is a block diagram illustrating an example of the configuration of a mixture ratio calculating unit 104. 
Fig. 60 is a diagram illustrating an example of an ideal mixture ratio a. 

Fig. 61 is a model diagram wherein the pixel values are developed over the time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 62 is a model diagram wherein the pixel values are developed over the time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 63 is a diagram describing approximation using correlation of foreground components. 
Fig. 64 is a diagram describing the relation between C, N, and P. 

Fig. 65 is a block diagram illustrating another configuration of the estimated mixture ratio processing unit 401 . 

Fig. 66 is a diagram illustrating an example of an estimated mixture ratio. 

Fig. 67 is a block diagram illustrating another configuration of a mixture ratio calculation unit 104. 

Fig. 68 is a flowchart explaining the processing for calculating mixture ratio. 

Fig, 69 is a flowchart describing processing for computing an estimated mixture ratio. 

Fig. 70 is a diagram describing a straight line approximating a mixture ratio a. 

Fig. 71 is a diagram describing a plane approximating a mixture ratio a. 

Fig. 72 is a diagram describing how pixels in multiple frames correspond at the time of calculating the mixture ratio a. 
Fig. 73 is a block diagram illustrating another configuration of the mixture ratio estimation processing unit 401 . 
Fig. 74 is a diagram illustrating an example of an estimated mixture ratio. 
Fig. 75 is a flowchart explaining the processing for calculating mixture ratio. 

Fig. 76 is a flowchart describing the processing for mixture ratio estimation by way of a model corresponding to a 
covered background region. 

Fig. 77 is a block diagram illustrating an example of the configuration of a foreground/background separation unit 
105. 

Fig. 78A is a diagram illustrating an input image, foreground region image, background region image, foreground 
component image, and background component image. 

Fig. 78B is a model diagram corresponding to an input image, foreground region image, background region image, 
foreground component image, and background component image. 

Fig. 79 is a model diagram wherein the pixel values are developed overthe time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 80 is a model diagram wherein the pixel values are developed overthe time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 81 is a model diagram wherein the pixel values are developed overthe time direction, and periods correspond- 
ing to shutter time are divided. 

Fig. 82 is a block diagram illustrating an example of the configuration of the separating unit 601 . 

Fig. 83 is a flowchart describing the processing for separating the foreground and the background. 

Fig. 84 is a block diagram illustrating the configuration of a separated image processing unit 106 which generates 

coefficient sets. 

Fig. 85 is a diagram explaining the relation between a tutor image and a student image. 

Fig. 86 is a block diagram illustrating the configuration of a learning unit 14. 

Fig. 87A is a diagram explaining class classification processing. 

Fig. 87B is a diagram explaining class classification processing. 

Fig. 88A is a diagram explaining ADRC processing. 

Fig. 88B is a diagram explaining ADRC processing. 

Fig. 89 is a diagram explaining coefficient sets which the separated image processing unit 106 generates. 

Fig. 90 is a flowchart explaining the learning processing for generating coefficient sets with the separated image 

processing unit 106. 

Fig. 91 is a flowchart for explaining the processing for generating coefficient sets corresponding to the background 
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region. 

Fig. 92 is a block diagram iilustrating the configuration of the separated image processing unit 1 06 which generates 

a higher resolution image in the spatial direction by executing class classification adaptation processing. 

Fig. 93 is a block diagram illustrating the configuration of a mapping unit 807. 

Fig. 94A is a diagram illustrating an example of an image in the mixed region of a tutor image. 

Fig. 94B is a diagram illustrating change in pixel values corresponding to the position in the spatial direction of an 

image in the mixed region of a tutor image. 

Fig. 95A is a diagram illustrating an example of an image in a mixed region, generated by conventional class 
classification adaptation processing. 

Fig. 95B is a diagram illustrating change in the pixel values corresponding to position in the spatial direction of an 
image in a mixed region, generated by conventional class classification adaptation processing. 
Fig. 96A is a diagram illustrating an example of an image in a mixed region, generated by the image processing 
device according to the present invention. 

Fig. 96B is a diagram illustrating change in the pixel values corresponding to the position in the spatial direction 

of a mixed region image, generated by the image processing device according to the present invention. 

Fig. 97A is a diagram illustrating an example of an image in a foreground region of a tutor image. 

Fig. 97B is a diagram illustrating change in pixel values of an image in the foreground region of a tutor image. 

Fig. 98A is a diagram illustrating an example of an image in a foreground region, generated by conventional class 

classification adaptation processing. 

Fig. 98B is a diagram illustrating change in the pixel values of an image in a foreground region, generated by 
conventional class classification adaptation processing. 

Fig. 99A is a diagram illustrating an example of an image in a foreground region, generated by the image processing 
device according to the present invention. 

Fig. 99B is a diagram illustrating change in the pixel values corresponding to position in the spatial direction of a 

foreground region image, generated by the image processing device according to the present invention. 

Fig. 1 00 is a flowchart explaining the processing for creating images with the separated image processing unit 1 06. 

Fig. 101 is a flowchart describing processing for predicting images corresponding to the background region. 

Fig. 102 is a block diagram illustrating the configuration of the separated image processing unit 1 06 wherein edge 

enhancing processing with difference effects is applied for each region. 

Fig. 103 is a block diagram illustrating the configuration of an edge enhancing unit 907. 

Fig. 104A is a diagram describing the processing for edge enhancement. 

Fig. 104B is a diagram describing the processing for edge enhancement. 

Fig. 104C is a diagram describing the processing for edge enhancement. 

Fig. 105 is a diagram illustrating filter coefficients. 

Fig. 106 is a diagram explaining operation of a high pass filter 921 . 

Fig. 1 07 is a diagram illustrating filter coefficients. 

Fig. 108 is a diagram explaining operation of the highpass filter 921 . 

Fig. 109 is a block diagram illustrating another configuration of the edge enhancing unit 907. 

Fig. 110 is a diagram illustrating filter coefficients. 

Fig. 111 is a diagram explaining operation of a filter 941 . 

Fig. 112 is a diagram illustrating filter coefficients. 

Fig. 113 is a diagram explaining operation of the filter 941 . 

Fig. 114 is a diagram explaining the processing by the separated image processing unit 106. 

Fig. 115 is a flowchart explaining the processing of edge enhancement processing with the separated image 

processing unit 106. 

Fig. 116 is a block diagram illustrating another configuration of the functions of the image processing device. 
Fig. 11 7 is a block diagram illustrating an example of the configuration of a mixture ratio calculating unit 11 01 . 
Fig. 1 1 8 is a block diagram illustrating an example of the configuration of a foreground/background separation unit 
1102. 

Fig. 119 is a block diagram illustrating another configuration of the functions of the image processing device. 
Fig. 120 is a diagram illustrating the correlation between a divided image, and a model diagram wherein the pixel 
values of pixels are developed over the time direction. 

Fig. 121 is a diagram illustrating the correlation between an image wherein movement blurring has been removed, 
and a model diagram wherein the pixel values of pixels are developed over the time direction. 
Fig. 122 is a diagram describing processing of the image processing device according to the present invention. 
Fig. 123 is a flowchart explaining the processing of images with the image processing device according to the 
present invention. 

Fig. 1 24 is a block diagram illustrating an example of the configuration of a foreground/background separation unit 
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2001. 

Fig. 125A is a diagram illustrating an input image, foreground component image, and background component 
image. 

Fig. 125B is a model diagram corresponding to an input image, foreground component image, and background 
component image. 

Fig. 126 Is a model diagram wherein the pixel values are developed over the time direction, and periods corre- 
sponding to shutter time are divided. 

Fig. 127 is a model diagram wherein the pixel values are developed over the time direction, and periods corre- 
sponding to shutter time are divided. 

Fig. 128 is a model diagram wherein the pixel values are developed over the time direction, and periods corre- 
sponding to shutter time are divided. 

Fig. 129 is a block diagram illustrating an example of the configuration of a separating unit 2601 . 

Fig. 130A is a diagram illustrating an example of a separated foreground component image. 

Fig. 130B is a diagram illustrating an example of a separated background component image. 

Fig. 131 is a flowchart describing the processing for separating the foreground and the background. 

Fig. 132 is a block diagram illustrating an example of the configuration of a movement blurring removal unit 2002. 

Fig. 133 is a diagram describing increments of processing. 

Fig. 134 is a model diagram wherein the pixel values of foreground component image are developed over the time 
direction, and periods corresponding to shutter time are divided. 

Fig. 1 35 is a model diagram wherein the pixel values of foreground component image are developed over the time 
direction, and periods corresponding to shutter time are divided. 

Fig. 1 36 is a model diagram wherein the pixel values of foreground component image are developed over the time 
direction, and periods corresponding to shutter time are divided. 

Fig. 1 37 is a flowchart explaining processing for removing movement blurring contained in the foreground compo- 
nent image by the movement blurring removal unit 2002. 
Fig. 138 is a diagram illustrating a model of a background component image. 
Fig. 139 is a diagram illustrating a model of a corrected background component image. 

Fig. 140 is a block diagram illustrating the configuration of the movement-blurring-removed-image processing unit 
2004 which generates coefficient sets. 

Fig. 141 is a block diagram illustrating the configuration of a learning unit 3006. 

Fig. 142 is a diagram explaining coefficient sets which the movement-blurring-removed-image processing unit 
2004 generates. 

Fig. 143 is a flowchart explaining the learning processing for generating coefficient sets by the movement-blurring- 
removed-image processing unit 2004. 

Fig. 1 44 is a flowchart for explaining the processing for generating coefficient sets corresponding to the background 
component image. 

Fig. 145 is a block diagram illustrating the configuration of the movement-blurring-removed-image processing unit 
2004 which executes class classification adaptation processing and generates a higher resolution image in the 
spatial direction. 

Fig. 146 is a diagram illustrating a model of a foreground component image wherein movement blurring has been 
removed. 

Fig. 147 is a diagram illustrating a model of a foreground component image wherein movement blurring has been 
added. 

Fig. 148 is a block diagram illustrating the configuration of a mapping unit 3103. 

Fig. 149 is a flowchart explaining the processing for creating an image with regard to the movement-biurring- 
removed-image processing unit 2004. 

Fig. 150 is a flowchart describing processing for predicting images corresponding to the background component 
image. 

Fig. 151 is a block diagram illustrating the configuration of the movement-blurring-removed-image processing unit 
2004 wherein edge enhancing processing with difference effects is applied for each image. 
Fig. 152 is a diagram explaining the processing of the movement-blurring-removed-image processing unit 2004. 
Fig. 153 is a flowchart explaining the processing of edge enhancement processing with the movement-blurring- 
removed-image processing unit 2004. 

Fig. 154 is a block diagram illustrating another configuration of the functions of the image processing device. 
Fig. 155 is a block diagram illustrating an example of the configuration of a foreground/background separation unit 
501. 
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Best Mode for Carrying Out the Invention 

[0086] Fig. 1 0 is a block diagram which illustrates the configuration of an embodiment of the image processing device 
according to the present invention. A CPU (Central Processing Unit) 71 performs various types of processing following 
programs stored in ROM (Read Only Memory) 72, or a storage unit 78. RAM (Random Access Memory) 73 suitably 
stores programs for the CPU 71 to execute, data, and so forth. These CPU 71, ROM 72, and RAM 73 are mutually 
connected via a bus 74. 

[0087] The CPU 71 is also connected to an input/output interface 75 via the bus 74. The input/output interface 75 is 
connected to an input unit 76 such as a keyboard, mouse, microphone, or the like, and is connected to an output unit 
77 such as a display, speaker, or the like. The CPU 71 performs various types of processing corresponding to instruc- 
tions input from the input unit 76. The CPU 71 then outputs images, audio, or the like, which are obtained as a result 
of processing, to the output unit 77. 

[0088] The storage unit 78 connected to the input/output interface 75 comprises a hard disk, for example, and stores 
programs for the CPU 71 to execute and various types of data. A communication unit 79 communicates with external 
devices via the Internet or other networks. In this case of the example, the communication unit 79 serves as an obtaining 
unit which obtains output from a sensor. 

[0089] Also, an arrangement may be made wherein programs are obtained via the communication unit 79, and are 
stored in the storage unit 78. 

[0090] A drive 80 connected to the input/output interface 75 drives a magnetic disk 91 , optical disk 92, magneto- 
optical disk 93, semiconductor memory 94, or the like, in the event that those are mounted thereon, and obtains pro- 
grams and data stored therein. The obtained programs and data are transmitted to the storage unit 78 and stored 
therein, as necessary. 

[0091] Fig. 11 is a block diagram which illustrates the configuration of the functions of the image processing device 
according to the present invention. 

[0092] Note that whether each function of the image processing device is realized by hardware or software does not 
matter. That is to say, each block diagram in the present Specification may be regarded as not only a hardware block 
diagram but also as a software function block diagram. 

[0093] Here, the input image input in the image processing device contains movement blurring. 
[0094] The movement blurring means distortion which is included in images corresponding to moving objects, which 
occurs due to movement of objects which are objects of image-taking in the real world and due to image-taking prop- 
erties of the sensor. 

[0095] in the present Specification, images corresponding to objects which are objects of image-taking in the real 
world are called image objects. 

[0096] Input images provided to the image processing device are provided to an object extracting unit 1 01 , a region 

specifying unit 103, a mixture ratio calculating unit 104, and a foreground/background separation unit 105. 

[0097] The object extracting unit 1 01 roughly extracts the image objects corresponding to the foreground object 

contained in the input image, and supplies the extracted image object to the movement detecting unit 102. The object 

extracting unit 1 01 roughly extracts the image object corresponding to the foreground object, for example, by detecting 

the outlines of the image object corresponding to the foreground object contained in input image. 

[0098] The object extracting unit 101 roughly extracts the image object corresponding to the background object 

contained in the input image, and supplies the extracted image object to the movement detecting unit 102. The object 

extracting unit 1 01 roughly extracts the image object corresponding to the background object, by the difference between 

the input image and the image object corresponding to the extracted foreground object, for example. 

[0099] Also, for example, an arrangement may be made wherein the object extracting unit 1 01 roughly extracts the 

image objects corresponding to the foreground objects and the image objects corresponding to the background objects 

based upon the difference between the background images stored in background memory provided therein and the 

input images. 

[01 00] The movement detecting unit 1 02 calculates the movement vectors of the image object corresponding to the 
roughly extracted foreground objects by techniques such as block matching, gradation, phase correlation, pixel recur- 
sion, or the like, and supplies the calculated movement vectors and movement vector position information (information 
for specifying the pixel positions corresponding to the movement vectors) to the region specifying unit 103. 
[0101] The movement vector output from the movement detecting unit 102 includes information corresponding to a 
movement amount v. 

[01 02] . Also, for example, an arrangement may be made wherein the movement detecting unit 1 02 outputs the move- 
ment vector per image object to the movement blurring adjustment unit 106 along with the pixel position information 
for specifying a pixel of the image object. 

[0103] The movement amount v is a value which represents the change of position of the image corresponding to 
the moving object in increments of pixel interval. For example, in the event that the object image corresponding to the 
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foreground moves so as to be displayed at a position four pixels away in the following frame with a given frame as a 
reference, the movement amount v of the image of the object corresponding to the foreground is 4. 
[0104] The region specifying unit 103 classifies each pixel of the input image into one of the foreground region, the 
background region, or the mixed region which consists of the uncovered background region and the covered back- 

s ground region, and supplies the information which indicates which of the foreground region, the background region, 
or the mixed region which consists of the uncovered background region or the covered background region, each pixel 
belongs to, (which will be referred to as region information hereafter), to the mixture ratio calculation unit 104 and the 
foreground/background separation unit 105. Note that details of the mixed region, uncovered background region, and 
the covered background region will be described later 

10 [0105] The mixture ratio calculating unit 104 calculates the mixture ratio corresponding to the pixels contained in the 
mixed region (which will be referred to as mixture ratio a hereafter) based upon the input image and the region infor- 
mation supplied from the region specifying unit 103, and supplies the calculated mixed ratio to the foreground/back- 
ground separating unit 1 05. 

[0106] The mixture ratio a is a.value which represents the ratio of the image component corresponding to the back- 
's ground object (which will also be referred to as background component hereafter) with the pixel value as indicated in 
Expression (3) described below. 

[01 07] The foreground/background separation unit 1 05 separates the image components corresponding to the fore- 
ground object (which will be also referred to as the foreground component hereafter) and the background component 
image which consists of only the background components based upon the region Information supplied from the region 

20 specifying unit 103 and the mixture ratio a supplied from the mixture ratio calculation unit 104, and supplies the image 
in the background region, the image which consists of only the background components in the uncovered background 
region (which will be referred to as the background component image in the uncovered background region), the image 
which consists of only the foreground components in the uncovered background region (which will be referred to as 
the foreground component image in the uncovered background region), the image which consists of only the back- 

25 ground components in the covered background region (which will be referred to as the background component image 
in the covered background region), the image which consists of only the foreground components in the covered back- 
ground region (which will be referred to as the foreground component image in the covered background region), and 
the image in the foreground region, to the separated image processing unit 106. 

[0108] The separated image processing unit 106 performs processing for the image in the background region, the 
30 background component image in the uncovered background region, the foreground component image in the uncovered 
background region, the background component image in the covered background region, the foreground component 
image in the covered background region, and the image in the foreground region, supplied from the foreground/back- 
ground separation unit 105, respectively. 

[0109] For example, the separated image processing unit 106 generates coefficients which are used in the class 
35 classification adaptation processing for generating an even higher resolution image for each of the image in the back- 
ground region, background component image in the uncovered background region, foreground component image in 
the uncovered background region, background component image in the covered background region, foreground com- 
ponent image in the covered background region, and image in the foreground region. 

[0110] For example, the separated image processing unit 106 creates an even higher resolution image by applying 
40 the class classification adaptation processing for each of the image in the background region, background component 
image in the uncovered background region, foreground component image in the uncovered background region, back- 
ground component image in the covered background region, foreground component image in the covered background 
region, and image in the foreground region. 

[0111] Also, for example, the separated image processing unit 106 applies the processing for edge enhancement 
« with differing degrees by using a different coefficient for each of the image in the background region, the background 
component image in the uncovered background region, the foreground component image in the uncovered background 
region, the background component image in the covered background region, the foreground component image in the 
covered background region, and the image in the foreground region. 

[0112] The input images supplied to the image processing device will now be described, referring to Fig. 12 through . 
50 Fig. 27. 

[01 1 3] Fig. 1 2 is a diagram which describes image-taking with a sensor. The sensor comprises a CCD video camera 
or the like, for example, including a CCD (Charge-Coupled Device) area sensor which is a soiid-state image-taking 
device. An object 111 corresponding to the foreground in the real world moves between an object 112 corresponding 
to the background in the real world, and the sensor, for example, from the left side to the right side horizontally in the 
55 drawing. 

[01 1 4] The sensor takes images of the object 1 1 1 corresponding to the foreground with the object 1 1 2 corresponding 
to the background. The sensor outputs the taken images in increments of one frame. For example, the sensor outputs 
images of 30 frames per second. In this case, the exposure period of the sensor is 1/30 seconds. The exposure period 
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represents a period from the sensor beginning conversion of input light into electric charges, up to the end of conversion 
of input light to electric charges. The exposure period will be also referred to as a shutter period hereafter. 
[01 15] Fig. 1 3 is a diagram which describes an arrangement of pixels. In Fig. 1 3, A through I denote individual pixels. 
These pixels are arranged on a plane corresponding to the image. One detecting element corresponding to one pixel 
is disposed on the sensor. Upon the sensor taking images, one detecting element outputs pixel values corresponding 
to one pixel which makes up the image. For example, a position in the X direction of the detecting elements corresponds 
to a position in the horizontal direction on the image, and a position in the Y direction of the detecting elements corre- 
sponds to a position in the vertical direction on the image. 

[0116] As shown in Fig. 14, for example, the detecting element of the CCD converts the input light into charges for 
a period corresponding to the shutter period, and accumulates the converted charges. The quantity of charges is 
approximately proportional to the strength of the input light and the period during which the light is input. The detecting 
element adds the charges converted from the input light to the accumulated charges in the period corresponding to 
the shutter period. That is to say, the detecting element integrates the input light during the period corresponding to 
the shutter period, and accumulates the amount of charges corresponding to the integrated light. It can also be said 
that the detecting element has integrating effects with regard to time. 

[0117] The charges accumulated in the detecting element are converted into a voltage value by a circuit not shown 
in the drawings, which is further converted to pixel values such as digital data or the like, and is output. Accordingly, 
individual pixel values output from a sensor have values projected in one-dimensional space, which is from a result 
wherein a given portion having a spatial extension of the object corresponding to the foreground or the background, 
is integrated for the shutter period. 

[0118] The image processing device extracts valid information buried in output signals due to such accumulation 
operations of the sensor, such as the mixture ratio a, for example. 

[01 1 9] Fig. 1 5A and Fig. 1 5B are diagrams which describe the image which is obtained by taking image of the object 
corresponding to the moving foreground and the object corresponding to the still background. Fig. 15A illustrates the 
image which is obtained by taking image of the object corresponding to the foreground with movement and the object 
corresponding to the still background. With the example shown in Fig. 1 5A, the object corresponding to the foreground 
moves from the left to the right horizontally in the drawing. 

[0120] Fig. 15B is a model diagram wherein pixel values corresponding to one line of the image shown in Fig. 15A 
develop over the time direction. The horizontal direction in Fig. 15B corresponds to the spatial direction X in Fig. 15A. 
[0121] The pixel values of pixels in the background regions are made up of only the background components, i.e., 
the image components corresponding to the background objects. The pixel values of pixels in the foreground regions 
are made up of only the foreground components, i.e., the image components corresponding to the foreground objects. 
[0122] The pixel values of pixels in mixed regions are made up of the background components and the foreground 
components. Since the pixel values in the mixed region consists of the background components and the foreground 
components, the mixed region can also be said to be a distortion region. The mixed regions are further classified Into 
covered background regions and uncovered background regions. 

[0123] The covered background region is a mixed region at a position corresponding to a leading portion in the 
progress direction of the foreground object with regard to the foreground region, and accordingly is a region wherein 
the background components are covered by the foreground corresponding to elapsing of time. 
[01 24] Conversely, the uncovered background region is a mixed region at a position corresponding to a trailing portion 
in the progress direction of the foreground object with regard to the foreground region, and accordingly is a region 
wherein the background components emerge corresponding to elapsing of time. 

[0125] As described above, images including the foreground region, background region, covered background region, 
or uncovered background region, are input as input images to the region specifying unit 1 03, the mixture ratio calculating 
unit 104, and the foreground/background separation unit 105. 

[0126] Fig. 1 6 is a diagram which describes the background region, foreground region, mixed region, covered back- 
ground region, and uncovered background region, as described above. In the event of corresponding to the images 
shown in Fig. 15A, the background region is the still portion, the foreground region is the moving portion, the covered 
background region of the mixed region is the portion which changes from the background to the foreground, and the 
uncovered background region of the mixed region is the portion which changes from the foreground to the background. 
[0127] Fig. 1 7 is a model diagram wherein pixel values of the pixels arrayed adjacently in one line in the Image that 
has been taken of the objects corresponding to the still foregrounds and the objects corresponding to the still back- 
grounds, develop over the time direction. For example, pixels arrayed in one line in a screen may be selected, as pixels 
adjacently arrayed in one line. 

[0128] The pixel values F01 through F04 shown in Fig. 17 are pixel values of pixels corresponding to the still fore- 
ground object. The pixel values B01 through B04 shown in Fig. 17 are pixel values of pixels corresponding to the still 
background object. 

[0129] The vertical direction in Fig. 17 represents elapsing of time from the top to the bottom in the drawing. The 
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position of the upper side of the rectangle in Fig. 1 7 corresponds to the time at which the sensor begins conversion of 
the input light into charges, and the position of the lower side of the rectangle in Fig. 17 corresponds to the time at 
which the sensor ends the conversion of the input light into charges. That is to say, the distance from the upper side 
to the lower side of the rectangle in Fig. 1 7 corresponds to the shutter period. 
s [0130] An arrangement wherein the shutter period equals the frame interval will now be described below, by way of 
an example. 

[0131] The horizontal direction in Fig. 1 7 corresponds to the spatial direction X as described in Fig. 15A. More par- 
ticularly, shown by way of an example in Fig. 17, the distance from the left side of the rectangle denoted by "F01" to 
the right side of the rectangle denoted by * , B04 B in Fig. 1 7, is eight times long as the pixel pitch, that is to say, corresponds 
10 to the interval of eight continuous pixels. 

[0132] In the event that the foreground objects and the background objects keep still, the light input to the sensor is 
not altered during the period corresponding to the shutter period. 

[0133] Now, the period corresponding to the shutter period is divided into two or more periods of equal length. For 
example, in the event that the virtual dividing number is 4, the model diagram shown in Fig. 17 can be represented by 
15 the model shown in Fig. 1 8. The virtual dividing number is set corresponding to the movement amount v or the like of 
the object corresponding to the foreground within the shutter period. For example, corresponding to the movement 
amount v of 4, the virtual dividing number is 4, and the period corresponding to the shutter period is divided into 4 
periods. 

[01 34] The uppermost row in the drawing corresponds to the first of the divided periods from the shutter being opened . 
20 The second row from the top in the drawing corresponds to the second of the divided periods from the shutter being 
opened. The third row from the top in the drawing corresponds to the third of the divided periods from the shutter being 
opened. The fourth row from the top in the drawing corresponds to the fourth of the divided periods from the shutter 
being opened. 

[0135] The divided shutter period corresponding to the movement amount v is also referred to as a shutter period/ 
25 v hereafter. 

[0136] In the event that the object corresponding to the foreground keeps still, the foreground component F01/V 
equals the value in which the pixel value F01 is divided by the virtual dividing number, since the light input to the sensor 
is not altered. Similarly, in the event that the object corresponding to the foreground keeps still, the foreground com- 
ponent F02/v equals the value of the pixel value F02 being divided by the virtual dividing number, the foreground 
30 component F03/v equals the value of the pixel value F03 being divided by the virtual dividing number, and the foreground 
component F04/v equals the value of the pixel value F04 being divided by the virtual dividing number. 
[0137] In the event that the object corresponding to the background keeps still, the background component B01/v 
equals the value of the pixel value B01 being divided by the virtual dividing number, since the light input to the sensor 
is not altered. Similarly, in the event that the object corresponding to the background keeps still, the background com- 
35 ponent B02/v equals the value of the pixel value B02 being divided by the virtual dividing number, B03/v equals the 
value of the pixel value B03 being divided by the virtual dividing number, and B04/v equals the value of the pixel value 
B04 being divided by the virtual dividing number. 

[0138] That is to say, in the event that the object corresponding to the foreground keeps still, the foreground com- 
ponent F01/V corresponding to the first shutter period/v from the shutter opening, the foreground component F01/v 

40 corresponding to the second shutter period/v from the shutter opening, the foreground component F01 /v corresponding 
to the third shutter period/v from the shutter opening, and the foreground component F01/V corresponding to the fourth 
shutter period/v from the shutter opening, are the same value, since the light corresponding to the foreground object 
which is input to the sensor is not altered during the period corresponding to the shutter period. F02/v through F04/v 
have the same relationship as F01/v. 

45 [0139] In the event that the object corresponding to the background keeps still, the background component B01/v 
corresponding to the first shutter period/v from the shutter opening, the-background components B01/v corresponding 
to the second shutter period/v from the shutter opening, the background components B01/v corresponding to the third 
shutter period/v from the shutter opening, and the background components B01/v corresponding to the fourth shutter 
period/v from the shutter opening, are the same value, since the light corresponding to the background object which 

so is input to the sensor is not altered during the period corresponding to the shutter period. B02/v through B04/V have 
the same relationship. 

[0140] A case will now be described wherein the object corresponding to the foreground moves while the object 
corresponding to the background keeps still. 

[0141] Fig. 1 9 is a model diagram wherein pixel values of the pixels on one line including the covered background 
55 region develop over the time direction in the event that, the object corresponding to the foreground moves towards the 
right side in the drawing. In Fig. 19, the movement amount v of the foreground is 4. Since one frame is a short period, 
an assumption may be made that the object corresponding to the foreground is a rigid body, and moves at a constant 
velocity. 
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[0142] In Fig. 19, the object image corresponding to the foreground moves so as to be displayed at a position four 
pixels to the right in the following frame, with a given frame as a reference. 

[01 43] In Fig. 1 9, the left-most pixel through the fourth pixel from the left, belong to the foreground region. In Fig. 19, 
the fifth through the seventh pixels from the left belong to the covered background region of the mixed region. In Fig. 

5 1 9, the right-most pixel belongs to the background region. 

[0144] Since the object corresponding to the foreground moves so as to hide the object corresponding to the back- 
ground with elapsing of time, the components contained in the pixel values of the pixels which belong to the covered 
background region change from the background components to the foreground components at a certain point of the 
period corresponding to the shutter period. 

w [0145] For example, the pixel value M shown with a heavy frame in Fig. 19, is represented by Expression (1). 

M = B02/V + B02/V + F07/V + F06/v (1 ) 

15 [01 46] For example, since the fifth pixel from the left includes a background component corresponding to one shutter 
period/v and foreground components corresponding to the three shutter period/vs, the mixture ratio a of the fifth pixel 
from the left is 1/4. Since the sixth pixel from the left contains background components corresponding to the two shutter 
period/vs and foreground components corresponding to the two shutter period/vs, the mixture ratio a of the sixth pixel 
from the left is 1/2. Since the seventh pixel from the left includes background components corresponding to the three 

20 shutter period/vs and a foreground component corresponding to the one shutter period/v, the mixture ratio a of the 
seventh pixel from the left is 3/4. 

[0147] Since an assumption may be made that the object corresponding to the foreground is a rigid body and the 
foreground image moves at a constant velocity so as to be displayed at a position four pixels to the right in the following 
frame, the foreground component F07/v of the first shutter period/v from the shutter opening of the fourth pixel from 

25 the left in Fig. 19, for example, equals the foreground component corresponding to the second shutter period/v from 
the shutter opening of the fifth pixel from the left in Fig. 19. Similarly, the foreground component F07/v equals the 
foreground component corresponding to the third shutter period/v from the shutter opening of the sixth pixel from the 
left in Fig. 19, and the foreground component corresponding to the fourth shutter period/v from the shutter opening of 
the seventh pixel from the left in Fig. 19, respectively. 

30 [0148] Since an assumption may be made that the object corresponding to the foreground is a rigid body and that 
the foreground image moves at a constant velocity so as to be displayed at a point four pixels to the right in the following 
frame, the foreground component F06/v of the first shutter period/v from the shutter opening of the third pixel from the 
left in Fig. 19, for example, equals the foreground component corresponding to the second shutter period/v from the 
shutter opening of the fourth pixel from the left in Fig. 1 9. Similarly, the foreground component F06/v equals the fore- 

35 ground component corresponding to the third shutter period/v from the shutter opening of the fifth pixel from the left in 
Fig. 19, and the foreground component corresponding to the fourth shutter periocl/v from the shutter opening of the 
sixth pixel from the left in Fig. 1 9, respectively. 

[0149] Since an assumption may be made that the object corresponding to the foreground is a rigid body and the 
foreground image moves at a constant velocity so as to be displayed at a position four pixels to the right in the following 

40 frame, the foreground component F05/v of the first shutter period/v from the shutter opening of the second pixel from 
the left in Fig. 19, for example, equals the foreground component corresponding to the second shutter period/v from 
the shutter opening of the third pixel from the left in Fig. 19. Similarly, the foreground component F05/v equals the 
foreground component corresponding to the third shutter period/v from the shutter opening of the fourth pixel from the 
left in Fig. 1 9, and the foreground component corresponding to the fourth shutter period/v from the shutter opening of 

<s the fifth pixel from the left in Fig. 1 9, respectively. 

[0150] Since an assumption may be made that the object corresponding to the foreground is a rigid body and the 
foreground image moves at a constant velocity so as to be displayed at a position four pixels to the right in the following 
frame, the foreground component F04/v of the first shutter period/v from the shutter opening of the left-most pixel in 
Fig. 19, for example, equals the foreground component corresponding to the second shutter period/v from the shutter 

so opening of the second pixel from the left in Fig. 19. Similarly, the foreground component F04/v equals the foreground 
component corresponding to the third shutter period/v from the shutter opening of the third pixel from the left in Fig. 
1 9, and the foreground component corresponding to the fourth shutter period/v from the shutter opening of the fourth 
pixel from the left in Fig. 19, respectively. 

[0151] As described above, the foreground region corresponding to the moving object includes movement blurring, 
55 so this can be said to be a distorted region. 

[0152] Fig. 20 is a model diagram wherein the pixel values of the pixels on one line including the uncovered back- 
ground region develop over the time direction in the event that the foreground moves toward the right side in the 
drawing. In Fig. 20, the movement amount v of the foreground is 4. Since one frame is a short time, an assumption 
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may be made that the object corresponding to the foreground is a rigid body, and moves at a constant velocity. In Fig. 
20, the object image corresponding to the foreground moves to the right side by four pixels in the following frame with 
a given frame as a reference. 

[0153] In Fig. 20, the left-most pixel through the fourth pixel from the left, belong to the background region. In Fig. 
20, the fifth through the seventh pixels from the left belong to the mixed region of the uncovered background. In Fig. 
20, the right-most pixel belongs to the foreground region. 

[0154] Since the object corresponding to the foreground which has hidden the object corresponding to the back- 
ground moves so as to be removed from the front of the object corresponding to the background with elapsing of time, 
the components included in the pixel values of the pixels which belong to the uncovered background region change 
from the foreground components to the background components at a certain point in the period corresponding to the 
shutter period. 

[0155] For example, the pixel value M 1 indicated with a heavy frame in Fig. 20, is represented by Expression (2). 



[0156] For example, since the fifth pixel from the left includes the background components corresponding to the 
three shutter period/vs, and the foreground component corresponding to the one shutter period/v, the mixture ratio a 
of the fifth pixel from the left is 3/4. Since the sixth pixel from the left includes the background components corresponding 
to the two shutter period/vs and the foreground components corresponding to the two shutter period/vs, the mixture 
ratio a of the sixth pixel from the left is 1/2. Since the seventh pixel from the left includes the background component 
corresponding to the one shutter period/v and the foreground components corresponding to the three shutter period/ 
vs, the mixture ratio a of the seventh pixel from the left is 1/4. 

[0157] Further generalizing Expression (1 ) and Expression (2), the pixel value M is represented by Expression (3). 



[0158] Here, a denotes the mixture ratio. B denotes the pixel value of the background, and Fi/v denotes the foreground 
component. 

[0159] Since an assumption may be made that the object corresponding to the foreground is a rigid body and moves 
at a constant velocity, and the movement amount v is 4, for example, the foreground component F01/v of the first 
shutter period/v from the shutter opening of the fifth pixel from the left in Fig. 20 equals the foreground component 
corresponding to the second shutter period/v from the shutter opening of the sixth pixel from the left in Fig. 20. Similarly, 
F01/V equals the foreground component corresponding to the third shutter period/v from the shutter opening of the 
seventh pixel from the left in Fig. 20, and the foreground component corresponding to the fourth shutter period/v from 
the shutter opening of the eighth pixel from the left in Fig. 20, respectively. 

[0160] Since an assumption may be made that the object corresponding to the foreground is a rigid body and moves 
at a constant velocity, and the virtual dividing number is 4, the foreground component F02/v of the first shutter period/ 
v from the shutter opening of the sixth pixel from the left in Fig. 20, for example, equals the foreground component 
corresponding to the second shutter period/v from the shutter opening of the seventh pixel from the left in Fig. 20. 
Similarly, the foreground component F02/V equals the foreground component corresponding to the third shutter period/ 
v from the shutter opening of the eighth pixel from the left in Fig. 20. 

[01 61 ] Since an assumption may be made that the object corresponding to the foreground is an rigid body and moves 
at a constant velocity, and movement amount v is 4, the foreground component F03/v of the first shutter period/v from 
the shutter opening of the seventh pixel from the left in Fig. 20, for example, equals the foreground component corre- 
sponding to the second shutter period/v from the shutter opening of the eighth pixel from the left in Fig. 20. 
[0162] While a description has been made in the description of Fig. 18 through Fig. 20 wherein the virtual dividing 
number is 4, the virtual dividing number corresponds to the movement amount v. The movement amount v generally 
corresponds to the movement velocity of the object corresponding to the foreground. For example, in the event that 
the object corresponding to the foreground moves so as to be displayed at a position four pixels to the right in the 
following frame with a given frame as a reference, the movement amount v is 4. The virtual dividing number is 4 
corresponding to the movement amount v. Similarly, for example, in the event that the object corresponding to the 
foreground moves so as to be displayed at a position six pixels to the left in the following frame with a given frame as 
a reference, the movement amount v is 6, and the virtual dividing number is 6. 

[0163] Fig. 21 and Fig. 22 illustrate the relationship between the foreground region, the background region, and the 



M' = F02/V + F01/V + B26/V + B26/V 
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mixed region which consists of the covered background region or the uncovered background region, and the foreground 
components and the background components, corresponding to the divided shutter period. 

[0164] Fig. 21 illustrates an example wherein the pixels of the foreground region, the background region, and the 
mixed region, are extracted from the image including the foreground corresponding to the object which moves in front 
of the still background. The reference character A shown in Fig. 21 denotes an object which moves in front of the still 
background. In the example shown in Fig. 21 , the object corresponding to the foreground denoted by the reference 
character A moves horizontally with regard to the screen. 

[0165] The frame #n + 1 is the frame following the frame #n, and the frame #n + 2 is the frame following the frame 
#n + 1. 

[01 66] Fig. 22 illustrates a model wherein the pixels of the foreground region, the background region, and the mixed 
region are extracted from one of frame #n through frame #n + 2, and the pixel values of the extracted pixels are 
developed over the time direction, with the movement amount v at 4. 

[0167] Since the object corresponding to the foreground moves, the pixel values of the foreground region consist of 
four different foreground components corresponding to the period of shutter period/v. For example, the left-most pixel 
of the pixels of the foreground region shown in Fig. 22 consists of F01/v, F02/v, F03/v, and F04/v. That is to say, the 
pixels of the foreground region include movement blurring. 

[01 68] Since the object corresponding to the background keeps still, the light corresponding to the background input 
to the sensor is not altered during the period corresponding to the shutter period. In this case, the pixel values of the 
background region do not contain movement blurring. 

[0169] The pixel value of the pixel which belongs to the mixed region made up of the covered background region or 
the uncovered background region consists of the foreground components and the background components. 
[0170] Next, a model will be described wherein, in the event that the image corresponding to the object moves, the 
pixel values of the pixels which are arrayed adjacently in a single line on multiple frames, and at the same position in 
the frames, develop over the time direction. For example, in the event that the image corresponding to the object moves 
horizontally on the screen, the pixels arrayed in a single line can be selected as pixels arrayed adjacently in a single line. 
[0171] Fig. 23 is a model diagram wherein the pixel values of pixels arrayed adjacently in a single line on three frames 
of images which are taken of the object corresponding to the still background, and are at the same position in the 
frames, develop over the time direction. The frame #n is the frame following the frame #n - 1 , and the frame #n + 1 is 
the frame following the frame #n. Other frames are denoted in the same way. 

[0172] The pixel values of the B01 through B12 shown in Fig. 23 are the pixel values of the pixels corresponding to 
the object of the still background. Since the object corresponding to the background keeps still, the pixel values of the 
corresponding pixels do not change in the frame #n - 1 through the frame #n + 1 . For example, the pixels in the frame 
#n and the pixels in the frame #n + 1 at the position corresponding to the pixel having a pixel value B05 in the frame 
#n • 1 , have a pixel value BOS, respectively. 

[01 73] Fig. 24 is a model diagram wherein the pixel values of pixels arrayed adjacently in a single line on three frames 
of images taken of the object corresponding to the foreground which moves to the right side in the drawing with the 
object corresponding to the still background, and at the same position in the frames, develop over the time direction. 
The models shown in Fig. 24 includes the covered background region. 

[0174] Since an assumption may be made in Fig. 24 that the object corresponding to the foreground is a rigid body 
and moves at a constant velocity, and the foreground image moves so as to be displayed at a position four pixels to 
the right side in the following frame, the foreground movement amount v is 4, and the virtual dividing number is 4. 
[0175] For example, the foreground component of the first shutter period/v from the shutter opening of the left-most 
pixel of the frame #n - 1 in Fig. 24 is F12/V, the foreground component of the second shutter period/v from the shutter 
opening of the second pixel from the left in Fig. 24 is also F1 2/v. The foreground component of the third shutter period/ 
v from the shutter opening of the third pixel from the left in Fig. 24, and the foreground component of the fourth shutter 
period/v from the shutter opening of the fourth pixel from the left in Fig. 24, are F12/V. 

[0176] The foreground component of the second shutter period/v from the shutter opening of the left-most pixel in 
the frame #n - 1 in Fig. 24 is F1 1/v, and the foreground component of the third shutter period/v from the shutter opening 
of the second pixel from the left in Fig. 24 is also F11/v. The foreground component of the fourth shutter period/v from 
the shutter opening of the third pixel from the left in Fig. 24 is F11/v. 

[0177] The foreground component of the third shutter period/v from the shutter opening of the left-most pixel in the 
frame #n - 1 in Fig. 24 is F10/v, and the foreground component of the fourth shutter period/v from the shutter opening 
of the second pixel from the left in Fig. 24 is also F1 0/v. The foreground component of the fourth shutter period/v from 
the shutter opening of the left-most pixel in the frame #n - 1 in Fig. 24 is F09/v. 

[0178] Since the object corresponding to the background keeps still, the background component of the first shutter 
period/v from the shutter opening of the second pixel from the left in the frame #n - 1 in Fig. 24 is B01/v. The background 
components of the first and second shutter period/vs from the shutter opening of the third pixel from the left in the 
frame #n - 1 in Fig. 24 are B02/v. The background components of the first through third shutter period/vs from the 
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shutter opening of the fourth pixel from the left in the frame #n - 1 in Fig. 24 are B03/v. 

[0179] In the frame #n - 1 in Fig. 24, the left-most pixel belongs to the foreground region, and the second through 
fourth pixels from the left belong to the mixed region of the covered background region. 

[0180] The fifth through twelfth pixels from the left in the frame #n - 1 in Fig. 24 belong to the background region, 

5 and the pixel values thereof are B04 through B1 1 , respectively. 

[0181] The first through fifth pixels from the left in the frame #n in Fig. 24 belong to the foreground region. The 
foreground component of the shutter period/v in the foreground region in the frame #n, is one of F05/v through F12/v. 
[01 82] Since an assumption may be made that the object corresponding to the foreground is a rigid body and moves 
at a constant velocity, and the foreground image moves so as to be displayed at a position four pixels to the right side 

10 in the following frame, the foreground component of the first shutter period/v from the shutter opening of the fifth pixel 
from the left in the frame #n in Fig. 24 is F12/V, the foreground component of the second shutter period/v from the 
shutter opening of the sixth pixel from the left in Fig. 24 is also F12/V. The foreground component of the third shutter 
period/v from the shutter opening of the seventh pixel from the left in Fig. 24, and the foreground component of the 
fourth shutter period/v from the shutter opening of the eighth pixel from the left in Fig. 24, are F12/v. 

15 [0183] The foreground component of the second shutter period/v from the shutter opening of the fifth pixel from the 
left in the frame #n in Fig. 24 is F11/v, and the foreground component of the third shutter period/v from the shutter 
opening of the sixth pixel from the left in Fig. 24 is also F11/v. The foreground component of the fourth shutter period/ 
v from the shutter opening of the seventh pixel from the left in Fig. 24 is F11/v 

[01 84] The foreground component of the third shutter period/v from the shutter opening of the fifth pixel from the left 
20 in the frame #n in Fig. 24 is F1 ON, and the foreground component of the fourth shutter period/v from the shutter opening 
of the sixth pixel from the left in Fig. 24 is also F10/v. The foreground component of the fourth shutter period/v from 
the shutter opening of the fifth pixel from the left in the frame #n in Fig. 24 is F09/v. 

[0185] Since the object corresponding to the background keeps still, the background component of the first shutter 
period/v from the shutter opening of the sixth pixel from the left in the frame #n in Fig. 24 is B05/v. The background 
25 components of the first and second shutter period/vs from the shutter opening of the seventh pixel from the left in the 
frame #n in Fig. 24 are B06/v. The background components of the first through third shutter period/vs from the shutter 
opening of the eighth pixel from the left in the frame #n in Fig. 24 are B07/v. 

[01 86] In the frame #n in Fig. 24, the sixth through eighth pixels from the left belong to the mixed region of the covered 
background region. 

30 [0187] The ninth through twelfth pixels from the left in the frame #n in Fig. 24 belong to the background region, and 
the pixel values are BOS through B1 1 , respectively. 

[0188] The first through ninth pixels from the left in the frame #n + 1 in Fig. 24 belong to the foreground region. The 
foreground component of the shutter period/v in the foreground region in the frame #n + 1 is one of F01/v through F12/v. 
[01 89] Since an assumption may be made that the object corresponding to the foreground is a rigid body and moves 

35 at a constant velocity, and the foreground image moves so as to be displayed at a position four pixels to the right side 
in the following frame, the foreground component of the first shutter period/v from the shutter opening of the ninth pixel 
from the left in the frame #n + 1 in Fig. 24 is F1 2/v, and the foreground component of the second shutter period/v from 
the shutter opening of the tenth pixel from the left in Fig. 24 is also F12/v. The foreground component of the third shutter 
period/v from the shutter opening of the eleventh pixel from the left in Fig. 24, and the foreground component of the 

40 fourth shutter period/v from the shutter opening of the twelfth pixel from the left in Fig. 24, are F12/V. 

[01 90] The foreground component of the second shutter period/v from the shutter opening of the ninth pixel from the 
left in the frame #n + 1 in Fig. 24 is F11/v, and the foreground component of the third shutter period/v from the shutter 
opening of the tenth pixel from the left in Fig. 24 is also F1 1/v. The foreground component of the fourth shutter period/ 
v from the shutter opening of the eleventh pixel from the left in Fig. 24 is F11/V. 

45 [0191] The foreground component of the third shutter period/v from the shutter opening of the ninth pixel from the 
left in the frame #n + 1 in Fig. 24 is F1 0/v, and the foreground component of the fourth shutter period/v from the shutter 
opening of the tenth pixel from the left in Fig. 24 is also F1 0/v. The foreground component of the fourth shutter period/ 
v from the shutter opening of the ninth pixel from the left in the frame #n + 1 in Fig. 24 is F09/v. 
[0192] Since the object corresponding to the background keeps still, the background component of the first shutter 

50 period/v from the shutter opening of the tenth pixel from the left in the frame #n + 1 in Fig. 24 is B09/v. The background 
components of the first and second shutter period/vs from the shutter opening of the eleventh pixel from the left in the 
frame #n + 1 in Fig. 24 are B10/V. The background components of the first through third shutter period/vs from the 
shutter opening of the twelfth pixel from the left in the frame #n + 1 in Fig. 24 are B11/v. 

[0193] In the frame #n + 1 in Fig. 24, the tenth through twelfth pixels from the left side correspond to the mixed region 
55 which is the covered background region. 

[0194] Fig. 25 is a model diagram wherein the foreground components are extracted from the pixel values illustrated 
in Fig. 24. 

[0195] Fig. 26 is a model diagram wherein the pixel values of the pixels adjacently arrayed in a row in three frames 
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of the images which are taken of the foreground corresponding to the object which moves to the right side in the drawing 
with the still background, and are at the same position in the frames, develop over the time direction. In Fig. 26, the 
model diagram includes the uncovered background region. 

[0196] in Fig. 26, an assumption may be made that the object corresponding to the foreground is a rigid body, and 
moves at a constant velocity. Since the object co responding to the foreground moves so as to be displayed at a position 
four pixels to the right side in the following frame, the movement amount v is 4. 

[0197] For example, the foreground component of the first shutter period/v from the shutter opening of the left-most 
pixel in the frame #n - 1 in Fig. 26 is F13/v, and the foreground component of the second shutter period/v from the 
shutter opening of the second pixel from the left in Fig. 26 is also F13/v. The foreground component of the third shutter 
period/v from the shutter opening of the third pixel from the left in Fig. 26, and the foreground component of the fourth 
shutter period/v from the shutter opening of the fourth pixel from the left in Fig. 26, are F13/v. 
[0198] The foreground component of the first shutter period/v from the shutter opening of the second pixel from the 
left in the frame #n - 1 in Fig. 26 is F1 4/v, and the foreground component of the second shutter period/v from the shutter 
opening of the third pixel from the left in Fig. 26 is also F14/v. The foreground component of the first shutter period/v 
from the shutter opening of the third pixel from the left in Fig. 26 is F16/v. 

[0199] Since the object corresponding to the background keeps still, the background component of the second 
through fourth shutter period/vs from the shutter opening of the left-most pixel in the frame #n - 1 in Fig. 26 is B25/V. 
The background components of the third and fourth shutter period/vs from the shutter opening of the second pixel from 
the left in the frame #n - 1 in Fig. 26 are B26/v. The background component of the fourth shutter period/v from the 
shutter opening of the third pixel from the left in the frame #n - 1 in Fig. 26 is B27/v. 

[0200] lntheframe#n-1 in Fig. 26, the left-most pixel through the third pixel belong to a mixed region of the uncovered 
background region. 

[0201] The fourth through twelfth pixels from the left in the frame #n - 1 in Fig. 26 belong to the foreground region. 
The foreground component in the frame is one of F13/v through F24/v. 

[0202] The left-most pixel through the fourth pixel from the left in the frame #n in Fig. 26 belong to the background 
region, and the pixel values are B25 through B28; respectively. 

[0203] Since an assumption may be made that the object corresponding to the foreground is a rigid body and moves 
at a constant velocity, and the foreground image moves so as to be displayed at a position four pixels to the right side 
in the following frame, the foreground component of the first shutter period/v from the shutter opening of the fifth pixel 
from the left in the frame #n in Fig. 26 is F13/v, and the foreground component of the second shutter period/v from the 
shutter opening of the sixth pixel from the left in Fig. 26 is also F13/v, The foreground component of the third shutter 
period/v from the shutter opening of the seventh pixel from the left in Fig. 26, and the foreground component of the 
fourth shutter period/v from the shutter opening of the eighth pixel from the left in Fig. 26, are F13/v. 
[0204] The foreground component of the first shutter period/v from the shutter opening of the sixth pixel from the left 
in the frame #n in Fig. 26 is F14/v, and the foreground component of the second shutter period/v from the shutter 
opening of the seventh pixel from the left in Fig. 26 is also F14/v. The foreground component of the first shutter period/ 
v from the shutter opening of the eighth pixel from the left in Fig. 26 is F15/V. 

[0205] Since the object corresponding to the background keeps still, the background components of the second 
through fourth shutter period/vs from the shutter opening of the fifth pixel from the left in the frame #n in Fig. 26 are 
B29/V. The background components of the third and fourth shutter period/vs from the shutter opening of the sixth pixel 
from the left in the frame #n in Fig. 26 are B30/v. The background component of the fourth shutter period/v from the 
shutter opening of the seventh pixel from the left in the frame #n in Fig. 26 is B31/v. 

[0206] In the frame #n in Fig. 26, the fifth through seventh pixels from the left belong to the mixed region of the 
uncovered background region. 

[0207] The eighth through twelfth pixels from the left in the frame #n in Fig. 26 belong to the foreground region. The 
value corresponding to the period of the shutter period/v in the foreground region in the frame #n is one of F1 3/v through 
F20/V. 

[0208] The left-most pixel through the eighth pixel from the left in the frame #n + 1 in Fig. 26, belong to the background 
region, and the pixel values thereof are B25 through B32, respectively. 

[0209] Since an assumption may be made that the object corresponding to the foreground is a rigid body and moves 
at a constant velocity, and the foreground image moves so as to be displayed at a position four pixels to the right side 
in the following frame, the foreground component of the first shutter period/v from the shutter opening of the ninth pixel 
from the left in the frame #n + 1 in Fig. 26 is F13/v, and the foreground component of the second shutter period/v from 
the shutter opening of the tenth pixel from the left in Fig. 26 is also F1 3/v. The foreground component of the third shutter 
period/v from the shutter opening of the eleventh pixel from the left in Fig. 26, and the foreground component of the 
fourth shutter period/v from the shutter opening of the twelfth pixel from the left in Fig. 26, are F13/v. 
[0210] The foreground component of the first shutter period/v from the shutter opening of the tenth pixel from the 
left in the frame #n + 1 in Fig. 26 is F14/v, and the foreground component of the second shutter period/v from the shutter 
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opening of the eleventh pixel from the left in Fig. 26 is also F1 4/v. The foreground component of the first shutter period/ 
v from the shutter opening of the twelfth pixel from the left in Fig. 26 is F1 5/v. 

[0211] Since the object corresponding to the background keeps still, the background components of the second 
through fourth shutter period/vs from the shutter opening of the ninth pixel from the left in the frame #n +1 in Fig. 26 
are B33/v. The background components of the third and fourth shutter period/vs from the shutter opening of the tenth 
pixel from the left in the frame #n + 1 in Fig. 26 are B34/v. The background component of the fourth shutter period/v 
from the shutter opening of the eleventh pixel from the left in the frame #n + 1 in Fig. 26 is B35/v. 
[0212] In the frame #n + 1 in Fig. 26, the ninth through eleventh pixels from the left belong to the mixed region of the 
uncovered background region. 

[0213] The twelfth pixel from the left in the frame #n + 1 in Fig. 26 belongs to the foreground region. The foreground 
component of the shutter period/v in the foreground region in the frame #n + 1 is one of F13/V through F16/V. 
[0214] Fig. 27 is a model diagram of the image wherein the foreground components are extracted from the pixel 
values shown in Fig. 26. 

[0215] Fig. 28 is a diagram which illustrates how the input image divided into the image in the foreground region, the 
image in the background region, the foreground component image in the covered background region, the background 
component image in the covered background region, the foreground component image in the uncovered background 
region, and the background component image in the uncovered background region, correspond to a model diagram 
wherein the pixel values of pixels develop over the time direction. 

[0216] As shown in Fig. 28, the input image is classified into the foreground region, background region, covered 
background region, and uncovered background region, by the region specifying unit 103. The input image is separated 
into the image in the foreground region, the image in the background region, the foreground component image in the 
covered background region, the background component image in the covered background region, the foreground 
component image in the uncovered background region, the background component image in the uncovered back- 
ground region, by the foreground/background separation unit 1 05 based upon the specified regions, i.e., the foreground 
region, background region, covered background region, and uncovered background region, and the mixture ratio a 
detected by the mixture ratio calculation unit 104. 

[0217] The separated images, i.e., the image in the foreground region, the image in the background region, the 
foreground component image in the covered background region, the background component image in the covered 
background region, the foreground component image in the uncovered background region, the background component 
image in the uncovered background region, are processed, respectively. 

[0218] Fig. 29 is a diagram which illustrates an example of the image divided into the foreground region, the back- 
ground region, and the mixed region. The region specifying unit 103 specifies the foreground region, background 
region, and mixed region, of the input image. The image processing device can divide the input image into the image 
of the foreground region, image of the background region, and image of the mixed region, based upon the region 
information indicating the foreground region, background region, and mixed region. 

[0219] As shown in Fig. 30, the foreground/background separation unit 1 05 separates the image of the mixed region 
into the foreground component image and the background component image based upon the region information sup- 
plied from the region specifying unit 103 and the mixture ratio a supplied from the mixture ratio calculating unit 104. 
[0220] Fig. 31 illustrates the flowchart which describes the processing for an image with the image processing device 
according to the present invention. 

[0221] In Step S101, the region specifying unit 103 specifies the foreground region, background region, covered 
background region, and uncovered background region of the input image, based upon the movement vector and the 
position information thereof supplied from the movement detecting unit 1 02 and the input image. Details of the process- 
ing for region specifying will be described later. 

[0222] In Step S102, the mixture ratio calculating unit 104 calculates the mixture ratio a based upon the region 
information supplied from the region specifying unit 103 and the input image. Details of the processing of the mixture 
ratio calculating unit 104 calculating the mixture ratio a will be described later. 

[0223] In Step S1 03, the foreground/background separation unit 1 05 separates the input image into the image in the 
foreground region, the image in the background region, the foreground component image in the covered background 
region, the background component image in the covered background region, the foreground component image in the 
uncovered background, and the background component image in the uncovered background region, based upon the 
region information supplied from the region specifying unit 103 and the mixture ratio a supplied from the mixture ratio 
calculation unit 104. Details of the processing for separation of an image by the foreground/background separation 
unit 1 05 will be described later. 

[0224] In Step S 1 04, the separated image processing unit 1 06 performs image processing for each of the separated 
images, i.e., the image in the foreground region, the image in the background region, the foreground component image 
in the covered background region, the background component image in the covered background region, the foreground 
component image in the uncovered background region, and the background component image in the uncovered back- 
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ground region, and the processing ends. Details of the image processing performed by the separated image processing 
unit 106 will be described later. 

[0225] As described above, the image processing device according to the present invention separates the input 
image into the image in the foreground region, the image in the background region, the foreground component image 

5 in the covered background region, the background component image in the covered background region, the foreground 
component image in the uncovered background region, and the background component image in the uncovered back- 
ground region, and performs the image processing for each of the separated images, i.e., the image in the foreground 
region, the image in the background region, the foreground component image in the covered background region, the 
background component image in the covered background region, the foreground component image in the uncovered 

10 background region, and the background component image in the uncovered background region, which are separated. 
[0226] Fig. 32 is a block diagram which illustrates an example of the configuration of the region specifying unit 1 03. 
The region specifying unit 1 03, of which configuration is shown in Fig. 32, does not use the movement vectors. Frame 
memory 201 stores the input images in increments of one frame. In the event that the object of the processing is the 
frame #n, the frame memory 201 stores the frame #n - 2 which is two frames previous from the frame #n, the frame 

15 #n - 1 which is one frame previous from the frame #n, the frame #n, the frame #n + 1 which is one frame following the 
frame #n, and the frame #n + 2 which is two frames following the frame #n. 

[0227] A still/motion judgment unit 202-1 reads out the pixel value of the pixel in the frame #n + 2, which is at the 
same position as the position of the pixel on the image, which is the object of specifying the region in the frame #n, 
and the pixel value of the pixel in the frame #n + 1 , which is at the same position as the position of the pixel on the 

20 image, which is the object of specifying the region of the frame #n, from the frame memory 201 , and calculates the 
absolute value of the difference between the read out pixel values. The still/motion judgment unit 202-1 judges whether 
or not the absolute value of the difference between the pixel value in the frame #n + 2 and the pixel value in the frame 
#n + 1 is greater than the predetermined threshold value Th, and in the event that judgment is made that the absolute 
value of the difference is greater than the threshold value Th, the still/motion judgment unit 202-1 supplies the still/ 

25 motion judgment, indicating motion, to a region judgment unit 203-1 . In the event that judgment is made that the absolute 
value of the difference between the pixel value of the pixel in the frame #n + 2 and the pixel value of the pixel in the 
frame #n + 1 is equal to or less than the threshold value Th, the still/motion judgment unit 202-1 supplies the still/motion 
judgment, indicating "still", to the region judgment unit 203-1. 

[0228] A still/motion judgment unit 202-2 reads out the pixel value of the pixel in the frame #n + 1 , which is at the 

30 same position as the position of the pixel on the image, which is the object of specifying the region in the frame #n, 
and the pixel value of pixel which is the object in the frame #n from the frame memory 201 , and calculates the absolute 
value of the difference between the pixel values. The still/motion judgment un it 202-2 judges whether or not the absolute 
value of the difference between the pixel value in the frame #n + 1 and the pixel value in the frame #n is greater than 
the predetermined threshold value Th, and in the event that judgment is made that the absolute value of the difference 

35 between the pixel values is greater than the threshold value Th, the still/motion judgment indicating motion is supplied 
to the region judgment unit 203-1 and the region judgment unit 203-2. In the event that judgment is made that the 
absolute value of the difference between the pixel value of the pixel in the frame #n + 1 and the pixel value of the pixel 
in the frame #n is equal to or smaller than the threshold value Th, the still/motion judgment unit 202-2 supplies the still/ 
motion judgment, indicating "still", to the region judgment unit 203-1 and the region judgment unit 203-2. 

40 [0229] The still/motion judgment unit 202-3 reads out the pixel value of the pixel, which is the object of specifying 
the region in the frame #n, and the pixel value of the pixel in the frame #n - 1 , which is at the same position as the 
position on the image of the pixel, which is the object of specifying the region in the frame #n, from the frame memory 
201 , and calculates the absolute value of the difference between the pixel values. The still/motion judgment unit 202-3 
judges whether or not the absolute value of the difference between the pixel value in the frame #n and the pixel value 

^5 in the frame #n - 1 is greater than the predetermined value Th, and in the event that judgment is made that the absolute 
value of the difference between the pixel values is greater than the threshold value Th, the still/motion judgment indi- 
cating motion is supplied to the region judgment unit 203-2 and the region judgment unit 203-3. In the event that 
judgment is made that the absolute value of the difference between the pixel value of the pixel in the frame #n and the 
pixel value of the pixel in the frame #n - 1 is equal to or smaller than the threshold value Th, the still/motion judgment 

so unit 202-3 supplies the still/motion judgment indicating "still" to the region judgment unit 203-2 and the region judgment 
unit 203-3. 

[0230] The still/motion judgment unit 202-4 reads out the pixel value of the pixel in the frame #n - 1 at the same 
position as the position of the pixel on the image, which is the object of specifying the region in the frame #n, and the 
pixel value of the pixel in the frame #n - 2 at the same position as the position of the pixel on the image, which is the 
55 object of specifying the region in the frame #n, from the frame memory 201 , and calculates the absolute value of the 
difference between the pixel values. The still/motion judgment unit 202-4 judges whether or not the absolute value of 
the difference between the pixel value in the frame #n - 1 and the pixel value in the frame #n - 2 is greater than the 
predetermined threshold value Th, and in the event that judgment is made that the absolute value of the difference 
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between the pixel values is greater than the threshold value Th, the still/motion judgment indicating motion is supplied 
to the region judgment unit 203-3. In the event that judgment is made that the absolute value of the difference between 
the pixel value of the pixel in the frame #n - 1 and the pixel value of the pixel in the frame #n - 2 is equal to or smaller 
than the threshold value Th, the still/motion judgment unit 202-4 supplies the still/motion judgment indicating M still a to 

5 the region judgment unit 203-3. 

[0231] In the event that the still/motion judgment supplied from the still/motion judgment unit 202-1 indicates "still", 
and the still/motion judgment supplied from the still/motion judgment unit 202-2 indicates motion, the region judgment 
unit 203-1 judges that the pixel which is the object of specifying the region in the frame #n belongs to the uncovered 
background region, and sets the uncovered background region judgment flag corresponding to the judged pixel in the 

10 region, to "1 which indicates that the pixel belongs to the uncovered background region. 

[0232] in the event that the still/motion judgment supplied from the still/motion judgment unit 202-1 indicates motion, 
or the still/motion judgment supplied from the still/motion judgment unit 202-2 indicates still, the region judgment unit 
203-1 judges that the pixel which is the object of specifying the region in the frame #n does not belong to the uncovered 
background region, and sets the uncovered background region judgment flag corresponding to the judged pixel in the 

is region to "0", which indicates that the pixel does not belong to the uncovered background region. 

[0233] The region judgment unit 203-1 supplies the uncovered background region judgment flag which has been set 
to "1" or "0", as described above, to the judgment flag storing memory 204. 

[0234] in the event that the still/motion judgment supplied from the still/motion judgment unit 202-2 indicates "still", 
and the still/motion judgment supplied from the still/motion judgment unit 202-3 indicates "still", the region judgment 
20 unit 203-2 judges that the pixel which is the object of specifying the region in the frame #n belongs to the still region, 
and sets the still region judgment flag corresponding to the pixel judged in the region, to "1", which indicates that the 
pixel belongs to the still region. 

[0235] In the event that the still/motion judgment supplied from the still/motion judgment unit 202-2 indicates motion, 
or the still/motion judgment supplied from the still/motion judgment unit 202-3 indicates motion, the region judgment 
25 unit 203-2 judges that the pixel which is the object of specifying the region in the frame #n does not belong to the still 
region, and sets the still region judgment flag corresponding to the judged pixel in the region, to "0", which indicates 
that the pixel does not belong to the still region. 

[0236] The region judgment unit 203-2 supplies the still region judgment flag which has been set to "1" or "0" as 
described above, to judgment flag storing frame memory 204. 

30 [0237] In the event that the still/motion judgment supplied from the still/motion judgment unit 202-2 indicates motion , 
and the still/motion judgment supplied from the still/motion judgment unit 202-3 indicates motion, the region judgment 
unit 203-2 judges the pixel which is the object of specifying the region in the frame #n belongs to the moving region, 
and sets the moving region judgment flag corresponding to the judged pixel in the region, to "1", which indicates that 
the pixel belongs to the moving region. 

35 [0238] In the event that the still/motion judgment supplied from the still/motion judgment unit 202-2 indicates "still", 
or the still/motion judgment supplied from the still/motion judgment unit 202-3 indicates "still", the region judgment unit 
203-2 judges that the pixel which is the object of specifying the region in the frame #n does not belong to the moving 
region, and sets the moving region judgment flag corresponding to the judged pixel in the region, to "0", which indicates 
that the pixel does not belong to the moving region. 

40 [0239] The region judgment unit 203-2 supplies the moving region judgment flag which has been set to "1" or "0", 
to the judgment flag storing frame memory 204. 

[0240] In the event that the still/motion judgment supplied from the still/motion judgment unit 202-3 indicates motion, 
and the still/motion judgment supplied from the still/motion judgment unit 202-4 indicates "still", the region judgment 
unit 203-3 judges that the pixel which is the object of specifying the region in the frame #n belongs to the covered 
45 background region, and sets the covered background region judgment flag corresponding to the judged pixel in the 
region to "1", which indicates that the pixel belongs to the covered background region. 

[0241] In the event that the still/motion judgment supplied from the still/motion judgment unit 202-3 indicates "still", 
or the still/motion judgment supplied from the still/motion judgment unit 202-4 indicates motion, the region judgment 
unit 203-3 judges that the pixel which is the object of specifying the region in the frame #n does not belong to the 
50 covered background region, and sets the covered background region judgment flag corresponding to the judged pixel 
in the region to "O", which indicates that the pixel does not belong to the covered background region. 
[0242] The region judgment unit 203-3 supplies the covered background region judgment flag which has been set 
to "1 " or "0" as described above, to the judgment flag storing frame memory 204. 

[0243] The judgment flag storing frame memory 204 stores the uncovered background region judgment flag supplied 
55 from the region judgment unit 203-1 , the still region judgment flag supplied from the region judgment unit 203-2, the 
moving region judgment flag supplied from the region judgment unit 203-2, and the covered background region judg- 
ment flag supplied from the region judgment unit 203-3. 

[0244] The judgment flag storing frame memory 204 supplies the uncovered background region judgment flag, the 
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still region judgment flag, the moving region judgment flag, and the covered background region judgment flag, which 
are stored therein, to a synthesizing unit 205. The synthesizing unit 205 generates the region information which indi- 
cates which of the uncovered background region, the still region, the moving region, or the covered background region, 
each pixel belongs to, and supplies the information to judgment flag storing frame memory 206, based upon the un- 
5 covered background region judgment flag, the still region judgment flag, the moving region judgment flag, and the 
covered background region judgment flag, which are supplied from the judgment flag storing frame memory 204. 
[0245] The judgment flag storing frame memory 206 stores the region information supplied from the synthesizing 
unit 205, and also outputs the stored region information. 

[0246] An example for processing performed by the region specifying unit 1 03 will now be described with reference 

10 to Fig. 33 through Fig. 37. 

[0247] In the event that the object corresponding to the foreground moves, the position of the image corresponding 
to the object on the screen changes with each frame. As shown in Fig. 33, in the frame #n, the image corresponding 
to the object which is at the position indicated by Yn(x,y) is at the position Yn + 1 (x,y) in the following frame #n + 1 . 
[0248] Fig. 34 is a model diagram wherein the pixel values of pixels of the image corresponding to the foreground 

15 object, which are adjacently arrayed in sequence in a image movement direction, develop over the time direction. For 
example, in the event that the image moving direction corresponding to the foreground object is horizontal to the screen, 
the model diagram in Fig. 34 indicates the model wherein the pixel values of adjacent pixels in one line develop over 
the time direction. 

[0249] In Fig. 34, the line in the frame #n is the same as the line in the frame #n + 1 . 
20 [0250] The foreground components corresponding to the object, which are included in the second pixel through 
thirteenth pixel from the left in the frame #n, are included in the sixth through seventeenth pixels from the left in the 
frame #n + 1 . 

[0251] In the frame #n, the pixels belonging to the covered background region are the eleventh through thirteenth 
pixels from the left, and the pixels belonging to the uncovered background region are the second through fourth pixels 
25 from the left, in the frame #n + 1 , the pixels belonging to the covered background region are the fifteenth through 
seventeenth pixels from the left, and the pixels belonging to the uncovered background region are sixth through eighth 
pixels from the left. 

[0252] With the example shown in Fig. 34, the movement amount v is 4, since the foreground components included 
in the frame #n move by four pixels in the frame #n + 1 . The virtual dividing number is 4, corresponding to the movement 
30 value v. 

[0253] Next, a description will be made regarding the change of the pixel values of the pixels belonging to the mixed 
region in the frames previous to and following the frame of interest. 

[0254] In the frame #n wherein the background keeps still and the movement amount v of the foreground is 4, shown 
in Fig. 35, the pixels belonging to the covered background region are the fifteenth through seventeenth pixels from the 
35 left. Since the movement amount v is 4, in the previous frame #n - 1 , the fifteenth through seventeenth pixels from the 
left include only the background components, and belong to the background region. Also, in the frame # n - 2 which 
is one further before, the fifteenth through seventeenth pixels from the left contain only the background components, 
and belong to the background region. 

[0255] Note that since the object corresponding to the background keeps still, the pixel value of the fifteenth pixel 
40 from the left in the frame #n - 1 do not change from the pixel value of the fifteenth pixel from the left in the frame #n - 
2. Similarly, the pixel value of the sixteenth pixel from the left in the frame #n - 1 do not change from the pixel value of 
the sixteenth pixel from the left in the frame #n - 2, and the pixel values of the seventeenth pixel from the left in the 
frame #n - 1 do not change from the pixel value of the seventeenth pixel from the left in the frame #n - 2. 
[0256] That is to say, the pixels of the frame #n - 1 and frame #n - 2 corresponding to the pixels belonging to the 
45 covered background region in the frame #n consists of only the background components, and the pixel values do not 
change, and accordingly the absolute value of the difference therebetween is approximately zero. Accordingly, judg- 
ment is made that the still/motion judgment for the pixels of the frame #n - 1 and the frame #n - 2 corresponding to the 
pixels belonging to the mixed region in the frame #n is still by the still/motion judgment unit 202-4. 
[0257] Since the pixels belonging to the covered background region in the frame #n contain the foreground compo- 
50 nents, the pixel values are different from the case wherein the pixel values in the frame #n - 1 consist of only the 
background components. Accordingly, judgment is made that the still/motion judgment for the pixels belonging to the 
mixed region in the frame #n and the pixels in the frame #n - 1 corresponding thereto is motion by the still/motion 
judgment unit 202-3. 

[0258] As described above, the region judgment unit 203-3 judges that the corresponding pixels belong to the covered 
55 background region in the event that the still/motion judgment unit 202-3 supplies the results of the still/motion judgment 
which indicates motion, and the still/motion judgment unit 202-4 supplies the results of the still/motion judgment which 
indicates "still". 

[0259] In the frame #n wherein the background keeps still and the foreground movement amount v is 4 as shown in 
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Fig. 36, the pixels included in the uncovered background region are the second through fourth pixels from the left. 
Since the movement amount v is 4, in the frame #n + 1 following the frame #n, the second through fourth pixels from 
the left include only the background components, and belong to the background region. Also, in the frame #n + 2 further 
one frame following the frame #n + 1 , the second through fourth pixels from the left contain only the background 

s components, and belong to the background region. 

[0260] Note that since the object corresponding to the background keeps still, the pixel values of the second pixel 
from the left in the frame #n + 2 does not change from the pixel value of the second pixel from the left in the frame #n 
+ 1 . Similarly, the pixel value of the third pixel from the left in the frame #n + 2 does not change from the pixel value of 
the third pixel from the left in the frame #n + 1 , and the pixel value of the fourth pixel from the left in the frame #n + 2 

10 does not change from the pixel value of the fourth pixel from the left in the frame #n + 1 . 

[0261] That is to say, the pixels of the frame #n + 1 and the frame #n + 2, corresponding to the pixels belonging to 
the uncovered background region in the frame #n, consist of only the background components, so the pixel values 
thereof do not change, and accordingly the absolute value of the difference thereof is approximately zero. Accordingly, 
judgment is made that the still/motion judgment for the pixels of the frame #n + 1 and the frame #n + 2 corresponding 

*5 to the pixels belonging to the mixed region in the frame #n is "still" by the still/motion judgment unit 202-1 . 

[0262] Since the pixels belonging to the uncovered background region in the frame #n contain the foreground com- 
ponents, the pixel values are different from the case wherein the pixels consists of only the background components 
in the frame #n + 1 . Accordingly, judgment is made that the still/motion judgment for the pixels belonging to the mixed 
region in the frame #n and the pixels corresponding thereto in the frame #n + 1 is motion by the still/motion judgment 

20 unit 202-2. 

[0263] As described above, the region judgment unit 203-1 judges that the corresponding pixels belong to the un- 
covered background region in the event that the still/motion judgment unit 202-2 supplies the results of the still/motion 
judgment which indicates motion, and the still/motion judgment unit 202-1 supplies the still/motion judgment which 
indicates "still". 

25 [0264] Fig. 37 is a diagram which illustrates judgment conditions of the region specifying unit 103 in the frame #n. 
in the event that judgment is made that the pixel in the frame #n - 2 at the same position as the position of the pixel 
which is the object of judgment on the image in the frame #n, and the pixel in the frame #n - 1 at the same position as 
the position of the pixel which is the object of judgment on the image in the frame #n, are "still", and judgment is made 
that the pixel in the frame #n - 1 at the same position as the position ofthe pixel which is the object of judgment on the 

30 image in the frame #n, and the pixel in the frame #n are motion, the region specifying unit 103 judges that the pixel 
which is the object of judgment of the frame #n belongs to the covered background region. 

[0265] In the event that judgment is made that the pixel in the frame #n - 1 at the same position as the position of 
the pixel which is the object of judgment on the image in the frame #n, and the pixel in the frame #n, are judged to be 
"still", and judgment is made that the pixel in the frame #n and the pixel in the frame #n + 1 at the same position as 

35 the position of the pixel which is the object of judgment on the image in the frame #n, are judged to be "still", the region 
specifying unit 1 03 judges that the pixel which is the object of judgment of the frame #n belongs to the still region. 
[0266] In the event that judgment is made that the pixel in the frame #n - 1 at the same position as the position of 
the pixel which is the object of judgment on the image in the frame #n, and the pixel in the frame #n. are judged to be 
motion, and judgment is made that the pixel of the frame #n and the pixel in the frame #n + 1 at the same position as 

40 the position of the pixel which is the object of judgment on the image in the frame #n, are judged to be motion, the 
region specifying unit 1 03 judges that the pixel which is the object of judgment of the frame #n belongs to the movement 
region. 

[0267] In the event that judgment is made that the pixel of the frame #n and the pixel in the frame #n + 1 at the same 
position as the position of the pixel which is the object of judgment on the image in the frame #n, are motion, and 

45 judgment is made that the pixel in the frame #n + 1 at the same position as the position of the pixel which is the object 
of judgment on the image in the frame #n, and the pixel in the frame #n + 2 at the same position as the position of the 
pixel which is the object of judgment on the image in the frame #n, are judged to be "still", the region specifying unit 
1 03 judges that the pixel which is the object of judgment of the frame #n belongs to the uncovered background region. 
[0268] Fig. 38A through Fig. 38D are diagrams which illustrate examples of results of the region specifying unit 103 

so specifying the region. In Fig. 38A, the pixels which have been judged to belong to the covered background region are 
displayed in white. In Fig. 38B, the pixels which have been judged to belong to the uncovered background region are 
displayed in white. 

[0269] In Fig. 38C, the pixels which have been judged to belong to the movement region are displayed in white. In 
Fig. 38D, the pixels which have been judged to belong to the still region are displayed in white. 
55 [0270] Fig. 39 is a diagram which illustrates the region information as an image, indicating the mixed region of the 
region information which the judgment flag storing frame memory 206 outputs. In Fig. 39, the pixels which have been 
judged to belong to the covered background region or the uncovered background region, i.e., the pixels judged to 
belong to the mixed region, are displayed in white. The region information indicating the mixed region, which the judg- 
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ment flag storing frame memory 206 outputs, indicates the mixed region and the portions which have texture within 
the foreground region and are surrounded by portions which have no texture. 

[0271] Next, referring to the flowchart in Fig. 40, the processing for region specifying by the region specifying unit 
103 will be described. In Step S201 , the frame memory 201 obtains the images of the frame #n - 2 through the frame 

5 #n + 2, including the frame #n which is the object of judgment. 

[0272] In Step S202, the stillAnotion judgment unit 202-3 judges whether or not the pixel of the frame #n - 1 and the 
pixel of the frame #n at the same position keep still, and in the event of judgment of "still", the flow proceeds to Step 
S203, and the still/motion judgment unit 202-2 judges whether or not the pixel of the frame #n and the pixel of the 
frame #n + 1 at the same position keep still. 

w [0273] In Step S203, in the event that Judgment is made that the pixel of the frame #n and the pixel of the frame #n 
+ 1 at the same position are "still", the flow proceeds to Step S204, and the region judgment unit 203-2 sets the still 
region judgment flag corresponding to the judged pixel in the region to "1" which indicates the pixel belongs to the still 
region. The region judgment unit 203-2 supplies the still region judgment flag to the judgment flag storing frame memory 
204, and the procedure proceeds to Step S205. 

is [0274] In Step S202, in the event that judgment is made that the pixel of the frame #n - 1 and the pixel of the frame 
#n at the same position are motion, or in Step S203, judgment is made that the pixel of the frame #n and the pixel of 
the frame #n + 1 at the same position are motion, the pixel of the frame #n does not belong to the still region, and 
accordingly the processing in Step S204 is skipped, and the procedure proceeds to Step S205. 
[0275] In Step S205, the still/motion judgment unit 202-3 judges whether or not the pixel of the frame #n - 1 and the 

20 pixel of the frame #n at the same position are in motion, and in the event of judgment of motion, the flow proceeds to 
Step S206, and the still/motion judgment unit 202-2 judges whether or not the pixel of the frame #n and the pixel of 
the frame #n + 1 at the same position are in motion. 

[0276] In Step S206, in the event that judgment is made that the pixel of the frame #n and the pixel of the frame #n 
+ 1 at the same position are in motion, the flow proceeds to Step S207, the region judgment unit 203-2 set the movement 
25 region judgment flag corresponding to the judged pixel in the region to "1 n which indicates that the pixel belongs to the 
movement region. The region judgment unit 203-2 supplies the movement region judgment flag to the judgment flag 
storing frame memory 204, and the procedure proceeds to Step S208. 

[0277] In Step S205, in the event that judgment is made that the pixel of the frame #n - 1 and the pixel of the frame 
#n at the same position are "still", or in Step S206, in the event that judgment is made that the pixel of the frame #n 

30 and the pixel of the frame #n + 1 at the same position are "stilP, since the pixel of the frame #n does not belong to the 
movement region, the processing in Step S207 is skipped, and the procedure proceeds to Step S208. 
[0278] In Step S208, the still/motion judgment unit 202-4 judges whether or not the pixel of the frame #n - 2 and the 
pixel of the frame #n - 1 at the same position keeps still, and in the event of judgment of "still", the flow proceeds to 
Step S209, and the stiil/motion judgment unit 202-3 judges whether or not the pixel of the frame #n - 1 and the pixel 

35 of the frame #n at the same position are in motion. 

[0279] In Step S209, in the event that judgment is made that the pixel of the frame #n - 1 and the pixel of the frame 
#n at the same position are in motion, the flow proceeds to Step S210, and the region judgment unit 203-3 sets the 
covered background region judgment flag corresponding to the judged pixel in the region to "1" which indicates that 
the pixel belongs to the covered background region. The region judgment unit 203-3 supplies the covered background 

to region judgment flag to the judgment flag storing frame memory 204, and the procedure proceeds to Step S211 . 

[0280] In Step S208, in the event that judgment is made that the pixel of the frame #n -2 and the pixel of the frame 
#n - 1 at the same position are in motion , or in Step S209, in the event that judgment is made that the pixel of the frame 
#n - 1 and the pixel of the frame #n at the same position are "still", the pixel of the frame #n does not belong to the 
covered background region, so the processing in Step S210 is skipped, and the procedure proceeds to Step S211 . 

45 [0281] In Step S211 , the still/motion judgment unit 202-2 judges whether or not the pixel of the frame #n and the 
pixel of the frame #n + 1 at the same position are in motion, and in the event of judgment of motion, the flow proceeds 
to Step S212, and the still/motion judgment unit 202-1 judges whether or not the pixel of the frame #n + 1 and the pixel 
of the frame #n + 2 at the same position keep still. 

[0282] In Step S21 2, in the event that judgment is made that the pixel of the frame #n + 1 and the pixel of the frame 
50 #n + 2 at the same position are "still", the flow proceeds to Step S213, and the region judgment unit 203-1 sets the 
uncovered background region judgment flag corresponding to the judged pixel in the region to "1° which indicates that 
the pixel belongs to the uncovered background region. The region judgment unit 203-1 supplies the uncovered back- 
ground region judgment flag to the judgment flag storing frame memory 204, and the procedure proceeds to Step S21 4. 
[0283] In Step S21 1 , in the event that judgment is made that the pixel of the frame #n and the pixel of the frame #n 
55 + 1 at the same position are "still", or in Step 212, in the event that judgment is made that the pixel of the frame #n + 
1 and the pixel of the frame #n + 2 at the same position are in motion, since the pixel of the frame #n does not belong 
to the uncovered background region, the processing in Step S2 13 is skipped, and the procedure proceeds to Step S21 4. 
[0284] In Step S214, the region specifying unit 103 judges whether or not all the pixels in the frame #n are region- 
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specified, and in the event that judgment is made that not all pixels are region-specified, the procedure returns to Step 
S202, and repeats the processing of specifying the region for other pixels. 

[0285] In Step S21 4, in the event that judgment is made that all the pixels in the frame #n are region-specified, the 
flow proceeds to Step S215, and the synthesizing unit 205 generates the region information which indicates the mixed 
5 region based upon the uncovered background region judgment flag and the covered background region judgment flag, 
which are stored in the judgment flag storing frame memory 204, and furthermore generates the region information 
which indicates which of the uncovered background region, the stili region, the movement region, or the covered back- 
ground region, each pixel belongs to, sets the generated region information for the judgment flag storing frame memory 
206, and the processing ends. 

w [0286] As described above, the region specifying unit 1 03 can generate region information which indicates which of 
the movement region, the still region, the uncovered background region, or the covered background region, each pixel 
contained in the frame belongs to. 

[0287] Note that an arrangement may be made wherein the region specifying unit 103 generates the region infor- 
mation corresponding to the mixed region and the region information made up of flags which indicates which of the 
15 movement region, the stili region, or the mixed region, each of pixels contained in the frame belongs to, by applying 
the logical sum to the region information corresponding to the uncovered background region and the covered back- 
ground region. 

[0288] in the event that the object corresponding to the foreground has texture, the region specifying unit 1 03 can 
specify the movement region more accurately. 
20 [0289] The region specifying unit 1 03 can output the region information indicating the movement region as the region 
information indicating the foreground region, or output the region information indicating the still region as the region 
information indicating the background region. 

[0290] While description has been made wherein the object corresponding to the background keeps still, the process- 
ing of specifying the region described above can be applied even if the image corresponding to the background region 

25 contains motion. For example, in the event thatthe image corresponding to the background region moves in a constant 
manner, the region specifying unit 1 03 shifts the entire image corresponding to the movement, and performs processing 
in the same manner as with the case wherein the object corresponding to the background keeps still. Also, in the event 
that the image corresponding to the background region contains a different motion at each local position, the region 
specifying unit 103 selects the pixel corresponding to the motion, and performs the above-described processing. 

30 [0291] Fig. 41 is a block diagram which illustrates another example of the structure of the region specifying unit 1 03. 
The region specifying unit 103 shown in Fig. 41 does not use movement vectors. A background image generating unit 
301 generates the background image corresponding to the input image, and supplies the generated background image 
to a binary object image extracting unit 302. The background image generating unit 301 extracts, for example, the 
image object corresponding to the background object contained in the input image, and generates the background 

35 image. 

[0292] An example of a model diagram is illustrated in Fig. 42 wherein the pixel values of the pixels arrayed in 
sequence adjacently in a movement direction of the image corresponding to the foreground object develop over the 
time direction. For example, the model diagram in Fig. 42 illustrates a model wherein, in the event that the movement 
direction of the image corresponding to the foreground object is horizontal to the screen, the pixel values of the adjacent 

40 pixels in one line develop over the time direction. 

[0293] In Fig. 42, the line in the frame #n is the same as the line in the frame #n - 1 and the line in the frame #n + 1 . 
[0294] In the frame #n, the foreground components corresponding to the object, which are contained in the sixth 
pixel through seventeenth pixel from the left, are contained in the second through thirteenth pixels from the left in the 
frame #n - 1 , and are contained in the tenth through twenty first pixels from the left in the frame #n + 1 . 

45 [0295] In the frame #n - 1 , the pixels belonging to the covered background region are the eleventh through thirteenth 
pixels from the left, and the pixels belonging to the uncovered background region are the second through fourth pixels 
from the left. In the frame #n, the pixels belonging to the covered background region are the fifteenth through the 
seventeenth pixels from the left, and the pixels belonging to the uncovered background region are the sixth through 
eighth pixels from the left. In the frame #n + 1 , the pixels belonging to the covered background region are the nineteenth 

so through twenty first pixels from the left, and the pixels belonging to the uncovered background region are the tenth 
through twelfth pixels from the left. 

[0296] In the frame #n - 1 , the pixels belonging to the background region are the first from the left, and the fourteenth 
through twenty first pixels from the left. In the frame #n, the pixels belonging to the background region are the first 
through fifth pixels from the left, and the eighteenth through twenty first pixels from the left. In the frame #n + 1 , the 
55 pixels belonging to the background region are the first through ninth pixels from the left. 

[0297] An example of the background image corresponding to the example shown in Fig. 42, which is generated by 
the background image generating unit 301, is illustrated in Fig. 43. The background image is made up of the pixels 
corresponding to the background object, and does not contain image components corresponding to the foreground 
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object. 

[0298] The binary object image extracting unit 302 generates a binary object image based upon the correlation 
between the background image and the input image, and supplies the generated binary object image to a time change 
detecting unit 303. 

[0299] Fig. 44 is a block diagram which illustrates the configuration of the binary object image extracting unit 302. A 
correlation value computing unit 321 computes the correlation between the background image supplied from the back- 
ground image generating unit 301 and the input image, generates a correlation value, and supplies the generated 
correlation value to a threshold value processing unit 322. 

[0300] The correlation value computing unit 321 applies Expression (4) to a block 3x3 wherein X 4 is centered in 
the background image as shown in Fig. 45A, and a block 3x3 wherein Y 4 corresponding to the block in the background 
image is centered in the input image as shown in Fig. 45B, and calculates a correlation value corresponding to the Y 4 , 
for example. 

Correlation Value = — , 2 ==~ (4) 

J±(Xi-X) -±(Yi-Y) 



£xi 

X = -*f- (5) 

y= J ~— (6) 

[0301] The correlation value computing unit 321 supplies the correlation value calculated corresponding to each 
pixel as described above to the threshold value processing unit 322. 

[0302] Also, an arrangement may be made wherein the correlation value computing unit 321 , for example, applies 
Expression (7) to the block 3 x 3 in the background image wherein X 4 is centered as shown in Fig. 46A, and the block 
3 x 3 in the input image wherein Y 4 is centered corresponding to the block in the background image as shown in Fig. 
46B, and calculates the sum of absolute value of difference corresponding to Y 4 . 



Sum of Absolute Value 
of Difference 




[0303] The correlation value computing unit 321 supplies the difference absolute value calculated as described above 
as the correlation value to the threshold value processing unit 322. 

[0304] The threshold value processing unit 322 compares the pixel value of the correlation image with the threshold 
value thO, and in the event that the correlation value is equal to or less than the threshold value thO, the threshold value 
processing unit 322 sets the pixel value of the binary object image to 1 , and in the event that the correlation value is 
greater than the threshold value thO, the threshold value processing unit 322 sets the pixel value of the binary object 
image to 0, and outputs the binary object image of which each pixel value has been set to 0 or 1 . The threshold value 
processing unit 322 may store the threshold value thO beforehand, and may use the threshold value thO which is input 
externally. 

[0305] Fig. 47 is a diagram which illustrates an example of the binary object image corresponding to the model of 
the input image shown in Fig. 42. In the binary object image, a pixel value of a pixel having a high correlation with the 
background image is set to 0. 

[0306] Fig. 48 is a block diagram which illustrates the configuration of the time change detecting unit 303. Frame 



28 



EP 1 396 819 A1 



memory 341 stores the binary object images of the frame #n - 1 , frame #n, and frame #n + 1 , supplied from the binary 
object image extracting unit 302 at the point of judgment of the region for the pixel of the frame #n. 
[0307] A region judgment unit 342 judges the region for each pixel of the frame #n based upon the binary object 
images of the frame #n -1 , frame #n, and frame #n + 1 , which are stored in the frame memory 341 , generates the 
region information, and outputs the generated region information. 

[0308] Fig. 49 is a diagram which describes the judgment made by the region judgment unit 342. In the event that 
the pixel of interest of the binary object image of the frame #n is 0, the region judgment unit 342 judges the pixel of 
interest of the frame #n to belong to the background region. 

[0309] In the event that the pixel of interest of the binary object image of the frame #n is 1 , the corresponding pixel 
of the binary object image of the frame #n - 1 is 1 , and the corresponding pixel of the binary object image of the frame 
#n + 1 is 1 , the region judgment unit 342 judges the pixel of interest of the frame #n to belong to the foreground region. 
[0310] In the event that the pixel of interest of the binary object image of the frame #n is 1 , and the corresponding 
pixel of the binary object image of the frame #n - 1 is 0, the region judgment unit 342 judges the pixel of interest of the 
frame #n to belong to the covered background region. 

[0311] In the event that the pixel of interest of the binary object image of the frame #n is 1 , and the corresponding 
pixel of the binary object image of the frame #n + 1 is 0, the region judgment unit 342 judges the pixel of interest of 
the frame #n to belong to the uncovered background region. 

[031 2] Fig. 50 is a diagram which illustrates an example wherein the time change detecting unit 303 judges the binary 
object image corresponding to the model of the input image shown in Fig. 42. The time change detecting unit 303 
judges the first through fifth pixels from the left of the frame #n to belong to the background region since the corre- 
sponding pixels of the binary object image of the frame #n are 0. 

[031 3] The time change detecting unit 303 judges the sixth through ninth pixels from the left to belong to the uncovered 
background region since the pixels of the binary object image of the frame #n are 1 , and the corresponding pixels of 
the frame #n + 1 are 0. 

[0314] The time change detecting unit 303 judges the tenth through thirteenth pixels from the left to belong to the 
foreground region since the pixels of the binary object image of the frame #n are 1 , the corresponding pixels of the 
frame #n - 1 are 1 , and the corresponding pixels of the frame #n + 1 are 1 . 

[0315] The time change detecting unit 303 judges the fourteenth through seventeenth pixels from the left to belong 
to the covered background region since the pixels of the binary object image of the frame #n are 1 , and the corre- 
sponding pixels of the frame #n - 1 are 0. 

[0316] The time change detecting unit 303 judges the eighteenth through twenty first pixels from the left to belong 
to the background region since the corresponding pixels of the binary object image of the frame #n are 0. 
[0317] The processing of specifying the region by the region judgment unit 103 will be now described, referring to 
the flowchart shown in Fig. 51 . In Step S301 , the background image generating unit 301 of the region judgment unit 
103, for example, generates the background image by extracting the image object corresponding to the background 
object contained in the input image based upon the input image, and supplies the generated background image to the 
binary object image extracting unit 302. 

[0318] In Step S302, the binary object image extracting unit 302 computes the correlation value between the input 
image and the background image supplied from the background image generating unit 301 by the computation de- 
scribed referring to Fig. 45, for example. In Step S303, the binary object image extracting unit 302 computes the binary 
object image from the correlation value and the threshold value thO by comparing the correlation value with the threshold 
value thO, for example. 

[031 9] In Step S304, the time change detecting unit 303 performs processing of region judgment, and the processing 
ends. 

[0320] The processing of the region judgment corresponding to Step S304 will be described in detail, referring to the 
flowchart shown in Fig. 52. In Step S321 , the region judgment unit 342 of the time change detecting unit 303 judges 
whether or not the pixel of interest in the frame #n stored in the frame memory 341 is 0, and in the event that the 
judgment is made that the pixel of the interest in the frame #n is 0, the flow proceeds to Step S322, makes settings to 
the effect that the pixel of interest in the frame #n belongs to the background region, and the processing ends. 
[0321] In Step S321 , in the event that judgment is made that the pixel of interest in the frame #n is 1 , the flow proceeds 
to Step S323, and the region judgment unit 342 of the time change detecting unit 303 judges whether or not the pixel 
of interest in the frame #n stored in the frame memory 341 is 1 , and the corresponding pixel in the frame #n - 1 is 0, 
and in the event that judgment is made that the pixel of interest in the frame #n is 1 , and the corresponding pixel in the 
frame #n - 1 is 0, the flow proceeds to Step S324, makes settings to the effect that the pixel of interest in the frame #n 
belongs to the covered background region, and the processing ends. 

[0322] In Step S323, in the event that judgment is made that the pixel of interest in the frame #n is 0, or the corre- 
sponding pixel in the frame #n - 1 is 1 , the flow proceeds to Step S325, and the region judgment unit 342 of the time 
change detecting unit 303 judges whether or not the pixel of interest in the frame #n stored in the frame memory 341 
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is 1 , and the corresponding pixel in the frame #n + 1 is 0, and in the event that judgment is made that the pixel of 
interest in the frame #n is 1 , and the corresponding pixel in the frame #n + 1 is 0, the flow proceeds to Step S326, 
makes settings to the effect that the pixel of interest in the frame #n belongs to the uncovered background region, and 
the processing ends. 

5 [0323] In Step S325, in the event that judgment is made that the pixel of interest in the frame #n is 0, or the corre- 
sponding pixel in the frame #n + 1 is 1, the flow proceeds to Step S327, and the region judgment unit 342 of the time 
change detecting unit 303 sets the pixel of interest in the frame #n for the foreground region, and the processing ends. 
[0324] As described above, the region specifying unit 1 03 can specify which of the foreground region, the background 
region, the covered background region, or the uncovered background region, the pixel of the input image belongs to, 

io and can generate region information corresponding to the specified results. 

[0325] Fig. 53 is a block diagram which illustrates another configuration of the region specifying unit 1 03. The region 
specifying unit 1 03 shown in Fig. 53 uses the movement vector and the position information thereof, which are supplied 
from the movement detecting unit 1 02. Portions the same as those shown in Fig. 41 are denoted by the same reference 
numerals, and description thereof will be omitted. 

15 [0326] A robustification unit 361 generates a robustified binary object image based upon N frames of the binary 
object image supplied from the binary object image extracting unit 302, and outputs to the time change detecting unit 
303. 

[0327] Fig. 54 is a block diagram which describes the configuration of the robustification unit 361 . A movement 
compensation unit 381 compensates for the movement of the binary object image of N frames based upon the move- 
20 ment vector and the position information thereof supplied from the movement detecting unit 1 02, and outputs the binary 
object image which has been subjected to compensation of movement to a switch 382. 

[0328] The movement compensation of the movement compensation unit 381 will be described with reference to 
examples shown in Fig. 55 and Fig. 56. For example, when judging the region in the frame #n, in the event that there 
is input of the binary object images of the frame #n - 1 , the frame #n, and the frame #n + 1 , shown by way of the 
25 example in Fig. 55, the movement compensation unit 381 compensates for movement of the binary object image of 
the frame #n - 1 and the binary object image of the frame #n + 1 , based upon the movement vector supplied from the 
movement detecting unit 102, and supplies the binary object image which has been subjected to compensation of 
movement to the switch 382, as indicated in the example shown in Fig. 56. 

[0329] The switch 382 outputs the binary object image which has been subjected to movement compensation of the 
30 first frame, to the frame memory 383-1 , and outputs the binary object image which has been subjected to movement 
compensation of the second frame to the frame memory 383-2. 

[0330] Similarly, the switch 382 outputs each of the binary object images of which the third through N - 1 'th frames 
have been subjected to compensation for the movement to each of frame memory 383-3 through frame memory 383-(N 
- 1), respectively, and outputs the binary object image of which the Nth frame has been subjected to movement com- 

35 pensation to frame memory 383- N. 

[0331 ] The frame memory 383-1 stores the binary object image of which the first frame has been subjected to move- 
ment compensation, and outputs the stored binary object image to a weighting addition unit 384-1 . The frame memory 
383-2 stores the binary object image of which the second frame has been subjected to movement compensation, and 
outputs the stored binary object image to a weighting addition unit 384-2. 

40 [0332] Similarly, each of the frame memory 383-3 through the frame memory 383-(N - 1 ) stores each of the binary 
object images of which one of the third frame through N - 1th frame has been subjected to compensation for the 
movement, and outputs the stored binary object image to each of the weighing addition unit 384-3 through the weighing 
addition unit 384-(N - 1). The frame memory 383-N stores the binary object image of which Nth frame has been 
subjected to compensation for the movement, and outputs the stored binary object image to a weighing addition unit 

45 384-N. 

[0333] The weighing addition unit 384-1 multiplies the pixel value of the binary object image of which the first frame 
has been subjected to compensation for the movement supplied from the frame memory 383-1 by the predetermined 
weight w1 , and supplies to an accumulation unit 385. The weighing addition unit 384-2 multiplies the pixel value of the 
binary object image of the second frame which has been subjected to movement compensation supplied from the 

so frame memory 383-2 by the predetermined weight w2, and supplies to an accumulation unit 385. 

[0334] Similarly, each of the weighting addition unit 384-3 through the weighing addition unit 384-(N - 1) multiplies 
the pixel value of the binary object image of one of the third through N - 1th frames, which has been subjected to 
movement compensation supplied from one of the frame memory 383-3 through the frame memory 383-(N - 1 ) by one 
of the predetermined weights w3 through w(N - 1), and supplies to the accumulation unit 385. A weighing addition unit 

ss 384-N multiplies the pixel value of the binary object image of the N'th frame supplied from the frame memory 383 - N 
which has been subjected to movement compensation by the predetermined weight wN, and supplies to the accumu- 
lation unit 385. 

[0335] The accumulation unit 385 accumulates the pixel value corresponding to the binary object image, wherein 
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each of the first through Nth frames which has been subjected to movement compensation Is multiplied by one of the 
predetermined weights w1 through wN, and generates the binary object image by comparing the accumulated pixel 
value with the predetermined threshold value thO. 

[0336] As described above, the robustification unit 361 generates the robustified binary object image from the N 
5 frames of binary object images, and supplies to the time change detecting unit 303, so the region specifying unit 103 
of which the configuration is shown in Fig. 53 can specify the region more accurately as compared with the case shown 
in Fig. 41 , even if the input image contains noise. 

[0337] The processing for specifying the region of the region specifying unit 1 03 of which the configuration is shown 
in Fig. 53 will now be described, referring to the flowchart shown in Fig. 57. The processing in Step S341 through Step 
10 S343 is the same as Step S301 through Step S303 described in the flowchart shown in Fig. 51 , respectively, and 
accordingly, description thereof will be omitted. 

[0338] In Step S344, the robustification unit 361 performs processing for robustification. 

[0339] In Step S345, the time change detecting unit 303 performs processing for specifying the region, and the 
processing ends. Details of the processing in Step S345 are the same as the processing described with reference to 

'5 the flowchart shown in Fig. 52, so description thereof will be omitted. 

[0340] Referring to the flowchart shown in Fig. 58, processing of robustification corresponding to the processing in 
Step S344 shown in Fig. 57 will now be described in detail. In Step S361, the movement compensation unit 381 
performs movement compensation processing of the input binary object image based upon the movement vector and 
the position information thereof supplied from the movement detecting unit 1 02. In Step S362, one of the frame memory 

20 383-1 through the frame memory 383-N stores the binary object image, which has been subjected to movement com- 
pensation, supplied via the switch 382. 

[0341] In Step S363, the robustification unit 361 judges whether or not N binary object images are stored, and in the 
event that judgment is made that N binary object images have not been stored, the flow returns to Step S361 , and the 
robustification unit 363 repeats processing of compensation for movement of the binary object image, and processing 
25 of storing the binary object image. 

[0342] In Step S363, in the event that judgment is made that N binary object images stored, the flow proceeds to 
Step S364, and each of the weighting addition units 384-1 through 384-N multiplies each of N binary object images, 
by one of the weights w1 through wN for weighting. 

[0343] In Step S365, the accumulation unit 385 accumulates the N weighted binary object images. 
30 [0344] In Step S366, the accumulation unit 385 generates the binary object image from the accumulated image, by 
comparing with the predetermined threshold value th1 , for example, and the processing ends. 
[0345] As described above, the region specifying unit 1 03, of which the configuration is shown in Fig. 53, can generate 
region information based upon the robustified binary object image. 

[0346] As described above, the region specifying unit 1 03 can generate the region information which indicates which 
35 of the movement region, the still region, the uncovered background region, or the covered background region, each 
of the pixels contained in the frame belongs to. 

[0347] Fig. 59 is a block diagram which illustrates an example of the configuration of the mixture ratio calculating 
unit 1 04. An estimated mixture ratio processing unit 401 calculates estimated mixture ratio for each pixel by computation 
corresponding to a model of a covered background region based upon the input image, and supplies the calculated 
40 estimated mixture ratio to a mixture ratio determination unit 403. 

[0348] An estimated mixture ratio processing unit 402 calculates estimated mixture ratio for each pixel by computation 
corresponding to a model of the uncovered background region based upon the input image, and supplies the calculated 
estimated mixture ratio to the mixture ratio determination unit 403. 

[0349] Since an assumption may be made that the object corresponding to the foreground moves at a constant 
45 velocity within a shutter period, the mixture ratio a of a pixel belonging to the mixed region has a nature such as 
described below. That is to say, the mixture ratio a changes linearly corresponding to the change of the position of the 
pixel. Taking the change of the pixel position to be one-dimensional, the change of the mixture ratio a may be repre- 
sented by a straight line, and taking the change of the pixel position to be two-dimensional, the change of the mixture 
ratio a may be represented by a plane. 
so [0350] Note that the period of one frame is short, an assumption may be made that the object corresponding to the 
foreground is a rigid body, and moves at a constant velocity. 

[0351] In this case, the inclination of the mixture ratio a is inversely proportionate to the movement amount v of the 
foreground within the shutter period. 

[0352] An example of an ideal mixture ratio a is shown in Fig. 60. The inclination 1 of an ideal mixture ratio a in the 
55 mixed region may be represented by the reciprocal of the movement amount v. 

[0353] As shown in Fig. 60, an ideal mixture ratio a has a value of 1 in the background region, and has a value of 0 

in the foreground region, and has a value which exceeds 0 and is less than 1 in the mixed region. 

[0354] With the example shown in Fig. 61 , the pixel value C06 of the seventh pixel from the left in the frame #n may 
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be represented in Expression (8), using the pixel value P06 of the seventh pixel from the left in the frame #n * 1. 

C06 = B06lv + B06h + FOllv + F02lv 
= P06h + P06h + FOUv + F02lv 
= 2lvP06+£ t Fih (8) 



[0355] In Expression (8), the pixel value C06 is represented as the pixel value M of the pixel in the mixed region, 
and the pixel value P06 is represented as the pixel value B of the pixel in the background region. That is to say, the 
pixel value M of the pixel in the mixed region and the pixel value B of the pixel in the background region may be 
represented as in Expression (9) and Expression (10), respectively. 

M = C06 (9) 
B=P06 (10) 



[0356] In Expression (8), 2/v corresponds to the mixture ratio a. Since the movement amount v is 4, the mixture ratio 
a of the seventh pixel from the left in the frame #n is 0.5. 

[0357] As described above, Expression (3) indicating the mixture ratio a may be rewritten as with Expression (11) 
25 by reckoning the pixel value C in the frame #n of interest to be a pixel value in the mixed region, and reckoning the 
pixel value P in the frame #n - 1 previous to the frame #n to be a pixel value of the background region. 



C = ct.P + f (11) 

30 

[0358] In Expression (11), f is the sum of the foreground components contained in the pixel of interest, L,Fi/v. The 
variables included in Expression (11) are two, i.e., the mixture ratio a and the sum of the foreground components f. 
[0359] In the same way, Fig. 62 illustrates a model wherein the pixel values wherein the movement amount v is 4, 
and virtual dividing number is 4 in the uncovered background region, develop over the time direction. 
35 [0360] Expression (3) indicating the mixture ratio a may be represented as in Expression (1 2) with the pixel value C 
in the frame #n of interest as a pixel value in the mixed region, and with the pixel value N in the frame #n + 1 following 
the frame #n as a pixel value in the background region, in the same manner as the representation in the covered 
background region described above, in the uncovered background region. 

40 C=ct'N + f (12) 

[0361] Note that while description has been made with an assumption that the background object keeps still, Ex- 
pression (8) through Expression (1 2) may be applied by using the pixel values of the pixels at the positions correspond- 
45 ing to the background movement amount v, even if the background object moves. For example, in Fig. 61 , in the event 
that the movement amount v of the object corresponding to the background is 2, the virtual dividing number is 2, and 
the object corresponding to the background moves to the right side in the drawing, the pixel value B of the pixel in the 
background region in Expression (10) is the pixel value P04. 

[0362] Expression (11) and Expression (12) include two variables, respectively, and accordingly the mixture ratio a 
50 can not be obtained in this state. Here, images generally have great correlation spatially, the adjacent pixels have 
approximately the same value. 

[0363] Thus, since the foreground components have great correlation spatially, the mixture ratio is obtained by trans- 
forming the expression so as to obtain the sum of the foreground components from the previous or following frame. 
[0364] The pixel value Mc of the seventh pixel from the left in the frame #n in Fig. 63 may be represented In Expression 

55 (13). 
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Mc=4r*B06 + '!LFi!v (13) 

[0365] The first argument 2/v of the right side in Expression (13) corresponds to the mixture ratio a. The second 
argument of the right side in Expression (1 3) is represented as in Expression (1 4) using the pixel values in the following 
frame #n + 1 . 



£ l FiIv = &-f l Filv (14) 

i=JJ . i=7 

[0366] Here, an assumption may be made that Expression (1 5) holds, using the spatial correlation of the foreground 
components. 

F = F05 = F06 = F07 = F08 = F09 = F10 = F1 1 = F12 (15) 
[0367] Expression (14) may be rewritten as Expression (16) using Expression (15).. 



i=/y V 

= fi.±. F (16) 

[0368] As a result, p may be represented in Expression (17). 

p = 2/4 (17) 

[0369] In general, in the event that an assumption is made wherein the foreground components correlated to the 
mixed region are the same as shown in Expression (15), Expression (18) is formed by the relationship of the internal 
dividing ratio for all the pixel in the mixed region. 

P = 1 - a (18) 
[0370] In the event that Expression (18) holds, Expression (11) may develop as indicated in Expression (19). 

r+v-y 

= ct>P+(l-at)'*ZFilv 

= a-P+(i-a)>N < 19 > 

[0371] Similarly, in the event that Expression (1 8) holds, Expression (1 2) may develop as indicated in Expression (20). 
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r+v-i 

= Ol-N+(l-(X)-ZFi/v 

=a-N+(i-a)-p < 20 > 

10 [0372] In Expression (19) and Expression (20), since C, N, and P are known pixel values, the variable included in 
Expression (19) and Expression (20) is only the mixture ratio a. The relationship between C, N, and P in Expression 
(1 9) and Expression (20) is illustrated in Fig. 64. C is the pixel value of the pixel of interest in the frame #n for calculating 
the mixture ratio a. N is the pixel value of the pixel in the frame #n + 1 , of which the position in the spatial direction 
corresponds to that of the pixel of interest. P is the pixel value of the pixel in the frame #n - 1 , of which the position in 

*5 the spatial direction corresponds to that of the pixel of interest. 

[0373] Accordingly, since Expression (1 9) and Expression (20) include one variable each, the mixture ratio a can be 
calculated using the pixel values in three frames. The conditions for calculating an accurate mixture ratio a by solving 
Expression (19) and Expression (20) are that; the foreground components with regard to the mixed region are the 
same, that is to say, in the foreground image object which has been taken in the state of the foreground object being 

20 still, the pixel values of pixels of a number double the movement amount v, which are arrayed sequentially at the 
boundary of the image object, corresponding to the movement direction of the foreground object, are constant. 
[0374] As described above, the mixture ratio a of the pixel belonging to the covered background region is calculated 
by Expression (21), and the mixture ratio a belonging to the uncovered background region is calculated by Expression 
(22). 

25 

a = (C - N)/(P - N) (21) 



30 a = (C-P)/(N-P) (22) 

[0375] Fig. 65 is a block diagram which illustrates the configuration of the estimated mixture ratio processing unit 
401 . Frame memory 421 stores the input image in increments of frames, and supplies the frame following the frame 
which is input as an input image, to frame memory 422 and a mixture ratio computation unit 423. 
35 [0376] The frame memory 422 stores the input image in increments of frames, and supplies the frame following the 
frame supplied from the frame memory 421 , to the mixture ratio computation unit 423. 

[0377] Accordingly, in the event that the frame #n + 1 is input as an input image to the mixture ratio computation unit 
423, the frame memory 421 supplies the frame #n to the mixture ratio computation unit 423, and the frame memory 
422 supplies the frame #n - 1 to the mixture ratio computation unit 423. 

40 [0378] The mixture ratio computation unit 423 calculates the estimated mixture ratio of the pixel of interest by the 
computation represented in Expression (21) based upon the pixel value C of the pixel of interest in the frame #n, the 
pixel value N of the pixel in the frame #n + 1 wherein the spatial position thereof corresponds to that of the pixel of 
interest, and the pixel value P of the pixel in the frame #n - 1 wherein the spatial position thereof corresponds to that 
of the pixel of interest, and outputs the calculated estimated mixture ratio. For example, in the event that the background 

45 keeps still, the mixture ratio computation unit 423 calculates the estimated mixture ratio of the pixel of interest based 
upon the pixel value C of the pixel of interest in the frame #n, the pixel value N of the pixel-in the frame #n + 1 at the 
same position in the frame as the pixel of interest, and the pixel value P of the pixel in the frame #n - 1 at the same 
position in the frame as the pixel of interest, and outputs the calculated estimated mixture ratio. 
[0379] As described above, the estimated mixture ratio processing unit 401 can calculate the estimated mixture ratio 

so based upon the input image, and supply to the mixture ratio decision unit 403, 

[0380] Note that the processing of the estimated mixture ratio processing unit 402 is the same as that of the estimated 
mixture ratio processing unit 401 except for the processing wherein, while the estimated mixture ratio processing unit 
401 calculates the estimated mixture ratio of the pixel of interest by the computation represented in Expression (21), 
the estimated mixture ratio processing unit 402 calculates the estimated mixture ratio of the pixel of interest by the 

55 computation represented in Expression (22), and accordingly, description thereof will be omitted. 

[0381] Fig. 66 is a diagram which illustrates an example of the estimated mixture ratio calculated by the estimated 
mixture ratio processing unit 401 . The estimated mixture ratio shown in Fig. 66 indicates the results in a case wherein 
the foreground movement amount v corresponding to the object which moves at a constant velocity is 11 , for one line. 
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[0382] It can be understood that the estimated mixture ratio changes generally linearly in the mixed region, as shown 
in Fig. 60. 

[0383] Returning to Fig. 59, the mixture ratio decision unit 403 sets the mixture ratio a based upon the region infor- 
mation indicating which of the foreground region, the background region, the covered background region, or the un- 
5 covered background region, the pixel which is the object of calculation of the mixture ratio a belongs to, supplied from 
the region specifying unit 103. 

[0384] In the event that the pixel which is the object belongs to the foreground region, the mixture ratio decision unit 
403 sets the mixture ratio a to 0, in the event that the pixel which is the object belongs to the background region, sets 
the mixture ratio a to 1 , in the event that the pixel which is the object belongs to the covered background region, sets 
10 the mixture ratio a to the estimated mixture ratio supplied from the estimated mixture ratio processing unit 401 , and in 
the event that the pixel which is the object belongs to the uncovered background region, sets the mixture ratio a to the 
estimated mixture ratio supplied from the estimated mixture ratio processing unit 402. The mixture ratio decision unit 
403 outputs the mixture ratio a which has been set based upon the region information. 

[0385] Fig. 67 is a block diagram which illustrates another configuration of the mixture ratio calculating unit 1 04. A 
is selection unit 441 supplies the pixels belonging to the covered background region and the corresponding pixels in the 
following and previous frames, to an estimated mixture ratio processing unit 442, based upon the region information 
supplied from the region specifying unit 103. The selection unit 441 supplies the pixels belonging to the uncovered 
background region and the corresponding pixels in the previous and following frames, to an estimated mixture ratio 
processing unit 443, based upon the region information supplied from the region specifying unit 103. 
20 [0386] The estimated mixture ratio processing unit 442 calculates the estimated mixture ratio of the pixel of interest 
belonging to the covered background region by the computation represented in Expression (21) based upon the pixel 
values input from the selection unit 441 , and supplies the calculated estimated mixture ratio to a selection unit 444. . 
[0387] The estimated mixture ratio processing unit 443 calculates the estimated mixture ratio of the pixel of interest 
belonging to the uncovered background region by the computation represented in Expression (22) based upon the 
25 pixel values input from the selection unit 441 , and supplies the calculated estimated mixture ratio to the selection unit 
444. 

[0388] In the event that the pixel which is the object belongs to the foreground region, the selection unit 444 selects 
the estimated mixture ratio of 0, and sets for the mixture ratio a, and in the event that the pixel which is the object 
belongs to the background region, the selection unit 444 selects the estimated mixture ratio of 1, and sets for the 

30 mixture ratio a, based upon the region information supplied from the region specifying unit 103. In the event that the 
pixel which is the object belongs to the covered background region, the selection unit 444 selects the estimated mixture 
ratio supplied from the estimated mixture ratio processing unit 442, and sets for the mixture ratio a, and in the event 
that the pixel which is the object belongs to the uncovered background region, the selection unit 444 selects the esti- 
mated mixture ratio supplied from the estimated mixture ratio processing unit 443, and sets this for the mixture ratio 

35 a. The selection unit 444 outputs the mixture ratio a which has been selected and set based upon the region information. 
[0389] As described above, the mixture ratio calculating unit 104 having another configuration shown in Fig. 67 can 
calculate the mixture ratio a for each pixel contained in the image, and output the calculated mixture ratio a. 
[0390] Referring to the flowchart shown in Fig. 68, the processing for calculation of the mixture ratio a by the mixtu re 
ratio calculating unit 104 of which configuration is shown in Fig. 59 will be described. In Step S401 , the mixture ratio 

40 calculating unit 104 obtains the region information supplied from the region specifying unit 103. In Step S402, the 
estimated mixture ratio processing unit 401 performs processing of computation of the estimated mixture ratio by a 
model corresponding to the covered background region, and supplies the calculated estimated mixture ratio to the 
mixture ratio decision unit 403. Details of the processing for computation of the estimated mixture ratio will be described 
later with reference to the flowchart shown in Fig. 69. 

45 [0391] In Step S403, the estimated mixture ratio processing unit 402 performs the processing of the computation of 
the estimated mixture ratio by a model corresponding to the uncovered background region, and supplies the calculated 
mixture ratio to the mixture ratio decision unit 403. 

[0392] In Step S404, the mixture ratio calculating unit 1 04 judges whether or not the mixture ratio a has been esti- 
mated for the entire frame, and in the event that judgment is made that the mixture ratio a has not been estimated for 
so the entire frame, the flow returns to Step S402, and performs the processing of estimation of the mixture ratio a for 
the following pixel. 

[0393] In the event that judgment is made in Step S404 that the mixture ratio a has been estimated for the entire 
frame, the flow proceeds to Step S405, and the mixture ratio decision unit 403 sets the mixture ratio a based upon the 
region information which indicates which of the foreground region, the background region, the covered background 
55 region, or the uncovered background region, the pixel belongs to, supplied from the region specifying unit 103. In the 
event that the pixel which is the object belongs to the foreground region, the mixture ratio decision unit 403 sets the 
mixture ratio a to 0, in the event that the pixel which is the object belongs to the background region, sets the mixture 
ratio a to 1 , in the event that the pixel which is the object belongs to the covered background region, sets the mixture 
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ratio a to the estimated mixture ratio supplied from the estimated mixture ratio processing unit 401 , and in the event 
that the pixel which is the object belongs to the uncovered background region, sets the mixture ratio a to the estimated 
mixture ratio supplied from the estimated mixture ratio processing unit 402, and the processing ends. 
[0394] As described above, the mixture ratio calculating unit 1 04 can calculate the mixture ratio a which is the amount 
5 of features corresponding to each pixel based upon the region information supplied from the region specifying unit 1 03 
and the input image. 

[0395] The processing of calculation of the mixture ratio a by the mixture ratio calculation unit 104 of which config- 
uration is shown in Fig. 67 is the same as the processing described in the flowchart shown in Fig. 68, so description 
thereof will be omitted. 

10 [0396] The processing for mixture ratio estimation by a model corresponding to the covered background region, 
which corresponds to Step S402 in Fig. 68, will now be described with reference to the flowchart shown in Fig. 69. 
[0397] In Step S421 , the mixture ratio computation unit 423 obtains the pixel value C of the pixel of interest in the 
frame #n from the frame memory 421 . 

[0398] In Step S422, the mixture ratio computation unit 423 obtains the pixel value P of the pixel in the frame #n • 
'5 1 , which corresponds to the pixel of interest, from the frame memory 422. 

[0399] In Step S423, the mixture ratio computation unit 423 obtains the pixel value N of the pixel in the frame #n + 
1 , which corresponds to the pixel of interest contained in the input image. 

[0400] In Step S424, the mixture ratio computation unit 423 computes the estimated mixture ratio based upon the 
pixel value C of the pixel of interest in the frame #n, the pixel value P of the pixel in the frame #n - 1 , and the pixel 

20 value N of the pixel in the frame #n + 1 . 

[0401] In Step S425, the mixture ratio computation unit 423 judges whether or not the processing for computation 
of the estimated mixture ratio has been ended for the entire frame, and in the event that judgment is made that the 
processing for computation of the estimated mixture ratio has not been ended for the entire frame, the flow returns to 
Step S421 , and the processing for calculating of the estimated mixture ratio is repeated for the following pixel. 

25 [0402] In Step S425, in the event that judgment is made that the processing for computation of the estimated mixture 
ratio has been ended for the entire frame, the processing ends. 

[0403] As described above, the estimated mixture ratio processing unit 401 can compute the estimated mixture ratio 
based upon the input image. 

[0404] The processing of mixture ratio estimation by a model corresponding to the uncovered background region 
30 shown in Step S403 in Fig. 68 is the same as the processing indicated in the flowchart shown in Fig. 69, wherein 
expressions corresponding to a model of the uncovered background region are used, and accordingly description 
thereof will be omitted. 

[0405] Note that the estimated mixture ratio processing unit 442 and the estimated mixture ratio processing unit 443 
shown in Fig. 67 compute the estimated mixture ratio by performing the processing the same as the processing indicated 

35 jn the flowchart shown in Fig. 69, and accordingly description thereof will be omitted. 

[0406] Also, while description has been made with an assumption that the object corresponding to the background 
keeps still, the processing for obtaining the mixture ratio a described above may be applied even if the image corre- 
sponding to the background region contains movement. For example, in the event that the image corresponding to the 
background moves uniformly, the estimated mixture ratio processing unit 401 shifts the entire image corresponding to 
the background movement, and performs processing in the same manner as in the case wherein the object corre- 
sponding to the background keeps still. Also, in the event that the image corresponding to the background region 
contains the background movement which is different at each local position, the estimated mixture ratio processing 
unit 401 selects the pixels corresponding to the background movement as the pixels corresponding to the pixels be- 
longing to the mixed region, and performs the processing described above. 

« [0407] Also, an arrangement may be made wherein the mixture ratio calculating unit 104 performs only the processing 
of the mixture ratio estimation by a model corresponding to the covered background region for all pixels, and outputs 
the calculated estimated mixture ratio as the mixture ratio a. In this case, the mixture ratio a indicates the ratio of the 
background components with regard to the pixels belonging to the covered background region, and indicates the ratio 
of the foreground components with regard to the pixels belonging to the uncovered background region. The image 

50 processing device can obtain the mixture ratio a indicating the ratio of the background components with regard to the 
pixels belonging to the uncovered background region, by calculating the absolute value of the difference between the 
mixture ratio a calculated as described above and 1 , and setting the calculated absolute value for the mixture ratio a, 
with regard to the pixels belonging to the uncovered background region. 

[0408] Note that similarly, an arrangement may be made wherein the mixture ratio calculating unit 1 04 performs only 
55 the processing for the mixture ratio estimation by a model corresponding to the uncovered background region for all 
pixels, and outputs the calculated estimated mixture ratio as the mixture ratio a. 
[0409] Another processing of the mixture ratio calculation unit 1 04 will now be described. 

[0410] An expression wherein the mixture ratio a and the sum of the foreground components f are approximated 
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spatially can be formed, using the nature wherein the mixture ratio a changes linearly corresponding to the change of 
the pixel position due to the object corresponding to the foreground moving at a constant velocity within a shutter 
period. The mixture ratio a is calculated by solving the expression wherein the mixture ratio a and the sum of the 
foreground components f are approximated, using multiple sets of the pixel value of the pixel belonging to the mixed 
region and the pixel value of the pixel belonging to the background region. 

[0411] In the event that the change of the mixture ratio a is generally linearly, the mixture ratio a is represented in 
Expression (23). 

a = il + p (23) 

[0412] In Expression (23), i denotes the index in the spatial direction wherein the position of the pixel of interest is 
0. 1 is the inclination of the straight line of the mixture ratio a. p is the intercept of the straight line of the mixture ratio 
a, as well as the mixture ratio a of the pixel of interest. In Expression (23), while the index i is known, the inclination 1 
and the intercept p are unknown. 

[0413] The correlation between the index i, the inclination 1 , and the intercept p, is shown in Fig. 70. 

[0414] In Fig. 70 and Fig. 71 , white circles indicate the pixel of interest. In Fig. 70, solid circles indicate pixels near 

the pixel of interest. 

[0415] Multiple different mixture ratio a for multiple pixels are represented by two variables by approximating the 
mixture ratio a as in Expression (23). In the example shown in Fig. 70, five mixture ratios for five pixels are represented 
by two variables, i.e., the inclination 1 and the intercept p. 

[0416] in the event of approximating the mixture ratio a in a planner manner shown in Fig. 71 , taking the movement 
v corresponding to the two directions of the horizontal direction and the vertical direction of the image into consideration, 
the mixture ratio a is represented in Expression (24) by expanding Expression (23) onto a plane. 

a = jm + kq + p (24) 

[0417] In Expression (24), j is the index in the horizontal direction wherein the position of the pixel of interest is 0, 
and k is the index in the vertical direction, m is the inclination of the mixture ratio a in the horizontal direction, and q is 
the inclination of the plane of the mixture ratio a in the vertical direction, p is the intercept of the plane of the mixture 
ratio a. 

[0418] For example, in the frame #n shown in Fig. 61 , Expression (25) through Expression (27) hold with regard to 
C05 through C07, respectively. 

C05= <x05- B05/v + f05 (25) 



C06=a06.B06/v + f06 (26) 

C07 = a07 • B07/V + f07 (27) 

[0419] Making an assumption that the foreground components generally agree, i.e., that F01 through F03 are the 
same, and that F01 through F03 are written as Fc, Expression (28) holds. 

f(x) = (1-a(x)).Fc (28) 

[0420] In Expression (28), x denotes the position in the spatial direction. 

[0421] Rewriting a(x) as Expression (24), Expression (28) may be represented as Expression (29). 

f(x) = (1 - 0m + kq + p)) . Fc 
= j.(-m.Fc) + k.(-q.Fc)+((1 - p) ■ Fc) 
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= js + kt + u(29) (29) 
[0422] In Expression (29), (-m • Fc), (-q • Fc), and (1 -p) • Fc are rewritten as Expression (30) through Expression (32). 

s = -m • Fc (30) 



w 



t = -q • Fc 



(31) 



u = (1 - p) • Fc 



(32) 



is [0423] In Expression (29), j is the index in the horizontal direction wherein the position of pixel of interest is 0, and 
k is the index in the vertical direction. 

[0424] As described above, since an assumption is made that the object corresponding to the foreground moves at 
a constant velocity within a shutter period, and the components corresponding to the foreground generally agree, the 
sum of the foreground components is approximated in Expression (29). 
20 [0425] Note that in the event of approximating the mixture ratio a linearly, the sum of the foreground components 
may be represented in Expression (33). 



25 



f (x) = is + u (33) 

[0426] Rewriting the mixture ratio a and the sum of the foreground components in Expression (13) using Expression 
(24) and Expression (29), the pixel value M is represented in Expression (34). 



30 



M = (jm + kq + p) • B + js + kt + u 
= jB-m + kB-q + B.p + j.s + k.t + u 



(34) 



[0427] in Expression (34), the unknown variables are the six values of the inclination of the plane of the mixture ratio 
35 a in the horizontal direction, m, the inclination of the plane of the mixture ratio a in the vertical direction, q, the intercepts 
of the plane of the mixture ratio a, p, s, t, and u. 

[0428] Setting the pixel value M and pixel value B for the normal equation represented in Expression (34) corre- 
sponding to the pixels near the pixel of interest, the mixture ratio a is calculated by solving multiple normal equations 
wherein the pixel value M and the pixel value B have been set, by the least square method. 
40 [0429] For example, with the index j of the pixel of interest in the horizontal direction as 0, with the index.k of the 
pixel of interest in the vertical direction as 0, and setting the pixel value M or the pixel value B for the normal equation 
represented in Expression (34) with regard to 3 x 3 pixels near the pixel of interest, Expression (35) through Expression 
(43) are obtained. 

= (-1) • B. V1 • m + (-1) . B. V1 ■ q + B. 1(-1 • p + 

(-1)-8 + (-1)-t + u (35) 



M o,.i = (°) • B o,-i • m + M) • B o,-i ■ q + B o.-i * P + (°) 

•s + (-1).t+u (36) 



55 



M + V1 = ( +1 ) • B + vi • m + (-1) • B +V1 . q + B +V1 • p + 
(+1)-s + (-1)-t+u 



(37) 
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M -i,o = M) ' B -i,o ■ m + (0) • B. 1f0 • q + B. 10 ■ p + 

(-1)-s + (0).t + u (38) 

M 0 , 0 = (0) • B o0 • m + (0) • B 0 o • q + B 0 0 ■ p + (0) • s 

+ (0) . t + u (39) 

M + 1,0 = H) • B + 1.0 • m + (0) - B +1 0 • q + B +10 • p + 

(+1).s + (0)-t + u (40) 

M. 141 = (-1) • B. 1i+1 • m + (+1) • B. 1f+1 • q + B. 1+1 • p + 

(-1).s + (+1).t+'u (41) 

M 0 +1 = (0) . B 0+1 ■ m + (+1) • B 0 +1 • q + B 0 +1 • p + (0) 

. s + ( + 1).t + u (42) 

= (+1) * B +li+1 ■ m + (+1) . B +1+1 ■ q + B +1+1 - p + 

( + 1). s + (+1)-t + u (43) 



[0430] Since the index of the pixel of interest in the horizontal direction, j, is 0, and the index in the vertical direction, 
k, is 0, the mixture ratio a of the pixel of interest is equal to the value wherein j = 0 and k = 0, from Expression (24), i. 
e., the intercept p. 

[0431] Accordingly, the intercept p can be output as the mixture ratio a by calculating the inclination in the horizontal 
35 direction, m, the inclination in the vertical direction, q, the intercept p, s, t, and u, by the least square method, based 
upon the nine expressions of Expression (35) through Expression (43). 

[0432] More specific procedures for calculating the mixture ratio a by applying the least square method will now be 
described. 

[0433] Representing the index i and the index k with one index x, the relationship between the index i, the index k, 
40 and the index x, is represented in Expression (44). 

x=(j + 1)'3 + (k+1) (44) 

45 [0434] The inclination in the horizontal direction, m, the inclination in the vertical direction, q, the intercept p, s, t, and 
u, are represented by variables, wO, w1, w2, w3, w4, and w5, respectively, and jB, kB, B, j, k, and I are represented 
by aO, a1, a2, a3, a4, and a5, respectively. Taking the margin of error ex into consideration, Expression (35) through 
Expression (43) are rewritten as Expression (45), 

50 

Mx = ^aywy + ex (45) 

y = 0 



55 [0435] In Expression (45), x denotes one of the integers between 0 and 8. 
[0436] Expression (46) may be derived from Expression (45). 
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5 

ex = Mx -^ClyWy 

[0437] To apply the least square method, the sum of squares of margin of error E is defined as represented in Ex- 
pression (47). 



E = i 0 ex2 < 47 > 

15 

[0438] To minimize the margin of error, the partial derivative of the squared-sum of the margin of error E from the 
variable Wv should be 0. Here v is one of the integers between 0 through 5. Accordingly, wy is calculated so as to 
satisfy Expression (48). 



(48) 



[0439] Substituting Expression (46) for Expression (48), Expression (49) is obtained. 



t l (avi i ay>wy) = i l avMx (49) 

x=0 y~Q x=0 

35 

[0440] Applying, for example, the sweeping method (Gauss-Jordan elimination) and so forth, to the six expressions 
each of which is obtained by substituting one of the integers between 0 and 5 for v in Expression (49), wy is calculated. 
As described above, wO denotes the inclination in the horizontal direction, m, w1 denotes the inclination in the vertical 
direction, q, w2 denotes the intercept p, w3 denotes s, w4 denotes t, and w5 denotes u. 
40 [0441] As described above, the inclination in the horizontal direction, m, the inclination in the vertical direction, q, 
the intercept p, s, t, and u may be obtained by applying the least square method to an expression wherein the pixel 
value M and the pixel value B have been set. 

[0442] In the description corresponding to Expression (35) through Expression (43), while description has been made 

with the pixel value of the pixel included in the mixed region as M, and the pixel value of the pixel included in the 
45 background region as B, the normal equation needs to be formed for each case of the pixel of interest being included 

in the covered background region, and being included in the uncovered background region. 

[0443] For example, in a case of obtaining the mixture ratio a of the pixel included in the covered background region 

of the frame #n shown in Fig. 61 , the pixels of the frame #n, C04 through C08, and the pixel values of the pixels of the 

frame #n - 1 , P04 through P08, are set for the normal equation. 
so [0444] In a case of obtaining the mixture ratio a of the pixel included In the uncovered background region of the 

frame #n shown in Fig. 62, the pixels of the frame #n, C28 through C32 t and the pixel values of the pixels of the frame 

#n + 1 , N28 through N32, are set for the normal equation. 

[0445] Also, for example, in the event of calculating the mixture ratio a of the pixel included in the covered background 
region shown in Fig. 72, Expression (50) through Expression (58) shown below may be formed. The pixel value of the 
55 pixel for calculation of the mixture ratio a is Mc5. 

Met = (-1) • Bc1 • m + (-1) • Bc1 - q + Bc1 • p + (-1) • 



(46) 



25 
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s + (-1).t + u (50) 

5 Mc2 = (0) • Bc2 • m + (-1) . Bc2 . q +■ Bc2 • p + (0) - s 

+ {-1).t + u (51) 

10 Mc3 = (+1) . Bc3 • m + (-1) • Bc3 . q + Bc3 . p + (+1) - 

s + (-1)-t + u (52) 

, 5 Mc4 = (-1) • Bc4 • m + (0) • Bc4 . q + Bc4 . p + (-1) • 

s + (0) • t + u (53) 

20 Mc5 = (0) • Bc5 • m + (0) • Bc5 . q + Bc5 . p + (0) . s 

+ (0) • t + u (54) 

25 Mc6 = (+1) • Bc6 • m + (0) • Bc6 • q + Bc6 • p + (+1) • 

s + (0) . t + u (55) 



30 



40 



Mc7 = (-1) . Bc7 • m + (+1) - Bc7 ■ q + Bc7 ■ p + (-1) • 

s + (+l).t + u (56) 



35 Mc8 = (0) • Bc8 • m + (+1) ■ Bc8 . q + Bc8 . p + (0) . s 

+ (+1)-t+u (57) 



Mc9 = (+1) • Bc9 • m + (+1) • Bc9 - q + Bc9 • p + (+1) • 

s + (+1)-t + u (58) 



[0446] In the event of calculating the mixture ratio a of the pixel included in the covered background region in the 
45 frame #n, the pixel values Bc1 through Bc9 of the pixels in the background region in the frame #n • 1 corresponding 
to the pixels in the frame #n, are used in Expression (50) through Expression (58). 

[0447] In Fig. 72, white circles indicate the pixels which are regarded as backgrounds, and solid circles indicate the 
pixels which are regarded as pixels in the mixed region. 

[0448] In the event of calculating the mixture ratio a of the pixel included in the uncovered background region shown 
so in Fig. 72, Expression (59) through Expression (67) described below may be formed. The pixel value of the pixel for 
calculation of the mixture ratio a is Mu5. 

Mu1 = (-1) • Bu1 • m + (-1) ■ Bu1 . q + Bu1 • p + (-1) • 
55 s + (-1)-t + u (59) 
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Mu2 = (0) ■ Bu2 . m + (-1) • Bu2 • q + Bu2 • p + (0) • s 

+ (-1)-t+u (60) 

Mu3 = (+1) • Bu3 • m + (-1) • Bu3 • q + Bu3 • p + (+1) • 

s + (-1).t + u (61) 

Mu4 = (-1) • Bu4 • m + (0) • Bu4 • q + Bu4 • p + (-1) . 

s + (0) . t + u (62) 

Mu5 = (0) • Bu5 • m + (0) • Bu5 • q + Bu5 • p + (0) ■ s 

+ (0) - 1 + u (63) 

Mu6 = (+1) • Bu6 • m + (0) • Bu6 • q + Bu6 • p + (+1) • 

s + (0) • t + u (64) 

Mu7 = (-1) • Bu7 - m + (+1) • Bu7 . q + Bu7 ■ p + (-1) • 

s + (+1).t + u (65) 

Mu8 = (0) • Bu8 • m + (+1) ■ Bu8 • q + Bu8 . p + (0) • s 

+ (+1).t+u (66) 

Mu9 = (+1) - Bu9 - m + (+1) ■ Bu9 • q + Bu9 . p + (+1) • 

s + {+1).t + u (67) 



[0449] In the event of calculating the mixture ratio a of the pixel included in the uncovered background region in the 
frame #n, the pixel values Bu1 through Bu9 of the pixels in the background region in the frame #n + 1 corresponding 
to the pixels in the frame #n, are used in Expression (59) through Expression (67). 

[0450] Fig. 73 is a block diagram which illustrates the configuration of the estimated mixture ratio processing unit 
45 401 . The image input to the estimated mixture ratio processing unit 401 is supplied to a delay circuit 501 and an addition 
unit 502. 

[0451] The delay circuit 501 delays the input image by one frame, and supplies to the addition unit 502. At the point 
that the frame #n is input to the addition unit 502 as an input image, the delay circuit 501 supplies the frame #n - 1 to 
the addition unit 502. 

so [0452] The addition unit 502 sets the pixel values of the pixels near the pixel for calculation of the mixture ratio a, 
and the pixel values of the frame #n - 1, for the normal equation. For example, the addition unit 502 sets the pixel 
values Mc1 through Mc9, and the pixel values Bc1 through Bc9 for the normal equation based upon Expression (50) 
through Expression (58). The addition unit 502 supplies the normal equation for which the pixel values have been set, 
to a computation unit 503. 

55 [0453] The computation unit 503 obtains the estimated mixture ratio by solving the normal equation supplied from 
the addition unit 502 by the sweeping method or the like, and outputs the obtained estimated mixture ratio. 
[0454] As described above, the estimated mixture ratio processing unit 401 can calculate the estimated mixture ratio 
based upon the input image, and supply to the mixture ratio decision unit 403. 
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[0455] Note that the estimated mixture ratio processing unit 402 has the same configuration as the estimated mixture 
ratio processing unit 401 , and accordingly description thereof will be omitted. 

[0456] Fig. 74 is a diagram which illustrates an example of the estimated mixture ratio calculated by the estimated 
mixture ratio processing unit 401. Fig. 74 illustrates the estimated mixture ratio with regard to one line, wherein the 
5 movement v of the foreground corresponding to the object which moves at a constant velocity is 11 , and the results 
are calculated by the expression generated in increments of blocks 7X7 pixels. 

[0457] It can be understood that the estimated mixture ratio changes generally linearty in the mixed region as shown 
in Fig. 60. 

[0458] The mixture ratio decision unit 403 sets the mixture ratio based upon the region information indicating which 
10 of the foreground region, the background region, the covered background region, or the uncovered background region 
the pixel for calculation of the mixture ratio belongs to, supplied from the region specifying unit 101 . In the event that 
the pixel which is the object belongs to the foreground region, the mixture ratio decision unit 403 sets the mixture ratio 
to 0, in the event that the pixel which is the object belongs to the background region, sets the mixture ratio to 1 , in the 
event that the pixel which is the object belongs to the covered background region, sets the mixture ratio to the estimated 
is mixture ratio supplied from the estimated mixture ratio processing unit 401 , and in the event that the pixel which is the 
object belongs to the uncovered background region, sets the mixture ratio to the estimated mixture ratio supplied from 
the estimated mixture ratio processing unit 402. The mixture ratio decision unit 403 outputs the mixture ratio which is 
set based upon the region information. 

[0459] Referring to the flowchart shown in Fig. 75, the processing for calculation of the mixture ratio by the mixture 
20 ratio calculating unit 102 in a case that the estimated mixture ratio processing unit 401 has a configuration shown in 
Fig. 73 will be described. In Step S501 , the mixture ratio calculating unit 102 obtains the region information supplied 
from the region specifying unit 1 01 . In Step S502, the estimated mixture ratio processing unit 401 performs the process- 
ing of mixture ratio estimation, by a model corresponding to the covered background region, and supplies the estimated 
mixture ratio to the mixture ratio decision unit 403. Details of the processing for mixture ratio estimation will be described 
25 later with reference to the flowchart shown in Fig. 76. 

[0460] In Step S503, the estimated mixture ratio processing unit 402 performs the processing of mixture ratio esti- 
mation by a model corresponding to the uncovered background region, and supplies the estimated mixture ratio to the 
mixture ratio decision unit 403. 

[0461 ] In Step S504, the mixture ratio calculating unit 1 02 judges whether or not the mixture ratio has been estimated 
30 for the entire frame, and in the event that judgment is made that the mixture ratio has not been estimated for the entire 
frame, the flow returns to Step S502, and the processing of mixture ratio estimation for the following pixel is performed. 
[0462] In Step S504, in the event that judgment is made that the mixture ratio has been estimated for the entire 
frame, the flow proceeds to Step S505, and the mixture ratio decision unit 403 sets the mixture ratio based upon the 
region information which indicates which of the foreground region, the background region, the covered background 
35 region, or the uncovered background region the pixel of calculation of the mixture ratio belongs to, supplied from the 
region specifying unit 1 01 . In the event that the pixel which is the object belongs to the foreground region, the mixture 
ratio decision unit 403 sets the mixture ratio to 0, in the event that the pixel which is the object belongs to the background 
region, sets the mixture ratio to 1 , in the event that the pixel which is the object belongs to the covered background 
region, sets the mixture ratio to the estimated mixture ratio supplied from the estimated mixture ratio processing unit 
40 401 , and in the event that the pixel which is the object belongs to the uncovered background region, sets the mixture 
ratio to the estimated mixture ratio supplied from the estimated mixture ratio processing unit 402, and processing ends. 
[0463] As described above, the mixture ratio calculating unit 1 02 can calculate the mixture ratio a which is the amount 
of features corresponding to each pixel based upon the region information supplied from the region specifying unit 1 01 
and the input image. 

45 [0464] Using the mixture ratio a enables the separation of the foreground components and the background compo- 
nents contained in the pixel value while leaving the information of movement blurring contained in the image corre- 
sponding to the moving object. 

[0465] Also, synthesizing an image based upon the mixture ratio a enables creation of an image containing accurate 
movement blurring corresponding to the speed of the object which moves as if image taking of the real world had been 
50 performed again. 

[0466] The processing for the mixture ratio estimation by a model corresponding to the covered background region, 
which corresponds to Step S502 shown in Fig. 75, will now be described with reference to the flowchart shown in Fig. 76 . 
[0467] In Step S521 , the addition unit 502 sets the pixel values contained in the input image and the pixels contained 
the image supplied from the delay circuit 501 for the normal equation corresponding to a model of the covered back- 
55 ground region. 

[0468] In Step S522, the estimated mixture ratio processing unit 401 judges whether or not setting of the pixel which 
is the object has ended, and in the event that judgment is made that the setting for the pixel which is the object has 
not ended, the flow returns to Step S521, and the processing of setting of the pixel value for the normal equation is 
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FIG. 51 
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FIG. 121 
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FIG. 122 
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FIG. 123 
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